Switch commands to ConfigKit configuration
This commit is contained in:
11
docs/cli.md
11
docs/cli.md
@@ -12,8 +12,8 @@ defined in [Operations](operations.md).
|
|||||||
notarius help
|
notarius help
|
||||||
notarius --version
|
notarius --version
|
||||||
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
|
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
|
||||||
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
notarius config validate [--config path/to/config.yml] [--config-profile name] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
notarius pipelines list [--config path/to/config.yml] [--json]
|
notarius pipelines list [--config path/to/config.yml] [--config-profile name] [--json]
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
Running Notarius without arguments, or with **help**, **--help**, or **-h**,
|
Running Notarius without arguments, or with **help**, **--help**, or **-h**,
|
||||||
@@ -42,6 +42,7 @@ pipeline ID and **--input** are required.
|
|||||||
| Flag | Meaning |
|
| Flag | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). |
|
| **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). |
|
||||||
|
| **--config-profile name** | Select a ConfigKit configuration overlay. Empty and **default** select the composed base configuration. |
|
||||||
| **--input path** | Source input file to process. Required. |
|
| **--input path** | Source input file to process. Required. |
|
||||||
| **--output-dir path** | Override the configured output root for this run. |
|
| **--output-dir path** | Override the configured output root for this run. |
|
||||||
| **--json** | Write the successful run-result receipt as JSON to standard output. |
|
| **--json** | Write the successful run-result receipt as JSON to standard output. |
|
||||||
@@ -154,10 +155,10 @@ OPENROUTER_API_KEY=your-api-key \
|
|||||||
## config validate
|
## config validate
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
notarius config validate [--config path/to/config.yml] [--config-profile name] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
This command loads and validates a configuration. With **--pipeline**, it also
|
This command loads and validates the selected composed configuration. With **--pipeline**, it also
|
||||||
resolves that pipeline against the production module catalog. **--only** selects
|
resolves that pipeline against the production module catalog. **--only** selects
|
||||||
lanes during that resolution and requires **--pipeline**.
|
lanes during that resolution and requires **--pipeline**.
|
||||||
|
|
||||||
@@ -183,7 +184,7 @@ validation; it cannot run a provider-backed pipeline.
|
|||||||
## pipelines list
|
## pipelines list
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
notarius pipelines list [--config path/to/config.yml] [--json]
|
notarius pipelines list [--config path/to/config.yml] [--config-profile name] [--json]
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
This command lists configured pipeline IDs in sorted order. By default, it
|
This command lists configured pipeline IDs in sorted order. By default, it
|
||||||
|
|||||||
142
docs/config.md
142
docs/config.md
@@ -1,7 +1,7 @@
|
|||||||
# Configuration
|
# Configuration
|
||||||
|
|
||||||
This is the canonical reference for Notarius configuration. Configuration files
|
This is the canonical reference for Notarius configuration. Configuration files
|
||||||
are YAML and must declare version 4. They select pipelines and their modules;
|
are YAML and must declare version 5. They select pipelines and their modules;
|
||||||
the [CLI reference](cli.md) owns invocation syntax, and
|
the [CLI reference](cli.md) owns invocation syntax, and
|
||||||
[Operations](operations.md) owns run-state procedures.
|
[Operations](operations.md) owns run-state procedures.
|
||||||
|
|
||||||
@@ -19,12 +19,15 @@ The command fails if none of these paths provides a configuration file.
|
|||||||
For configuration values, precedence is:
|
For configuration values, precedence is:
|
||||||
|
|
||||||
1. built-in defaults;
|
1. built-in defaults;
|
||||||
2. the selected YAML file;
|
2. composed YAML sources, including imports;
|
||||||
3. supported operational environment variables; and
|
3. the selected ConfigKit profile; and
|
||||||
4. the CLI run overrides that apply to a command.
|
4. explicitly supplied configuration-valued run flags.
|
||||||
|
|
||||||
Environment variables do not provide a second configuration schema. They only
|
`--config-profile` selects a ConfigKit overlay from `profiles/<name>.yml`
|
||||||
override the fields listed below.
|
beneath the selected configuration root. Empty and `default` select no overlay.
|
||||||
|
Configuration profiles are distinct from PromptKit LLM profiles selected by
|
||||||
|
`llm_profile` or `--llm-profile`. The retired operational environment variables
|
||||||
|
have no effect; `NOTARIUS_CONFIG` remains a source-discovery input.
|
||||||
|
|
||||||
## Maintained Examples
|
## Maintained Examples
|
||||||
|
|
||||||
@@ -46,7 +49,7 @@ other than **version** is optional.
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **version** | integer | none | Required; must be 4. |
|
| **version** | integer or quoted string | none | Required; must be 5. Maintained files use `version: 5`. |
|
||||||
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
||||||
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
||||||
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
||||||
@@ -73,13 +76,26 @@ Selecting a chunk-plan or checkpoint directory does not configure filesystem
|
|||||||
access policy. Cache permissions come from deployment provisioning, process
|
access policy. Cache permissions come from deployment provisioning, process
|
||||||
umask, setgid inheritance, and default ACLs; see [Operations](operations.md#cache-provisioning).
|
umask, setgid inheritance, and default ACLs; see [Operations](operations.md#cache-provisioning).
|
||||||
|
|
||||||
|
Every nonempty relative configured destination or reference path resolves from
|
||||||
|
the canonical directory of the file that supplies its surviving declaration.
|
||||||
|
This includes imported fragments and selected configuration profiles. Built-in
|
||||||
|
relative destinations use the selected main configuration file's directory;
|
||||||
|
explicit `--output-dir` and `--debug-dir` values use the invocation working
|
||||||
|
directory. Absolute values remain unchanged.
|
||||||
|
|
||||||
|
The root file can use ConfigKit imports to compose disjoint definitions. Imports
|
||||||
|
and selected profiles remain under the configuration root; they do not merge or
|
||||||
|
append a pipeline's `steps` sequence. A profile replaces sequences as defined
|
||||||
|
by ConfigKit, so changing an individual ordered binding requires supplying the
|
||||||
|
whole replacement sequence.
|
||||||
|
|
||||||
## PromptKit Profiles
|
## PromptKit Profiles
|
||||||
|
|
||||||
The optional **promptkit** object selects one source of profile definitions and
|
The optional **promptkit** object selects one source of profile definitions and
|
||||||
may register one conventional local OpenAI-compatible backend:
|
may register one conventional local OpenAI-compatible backend:
|
||||||
|
|
||||||
~~~yaml
|
~~~yaml
|
||||||
version: 4
|
version: 5
|
||||||
|
|
||||||
promptkit:
|
promptkit:
|
||||||
profile_dir: ./profiles
|
profile_dir: ./profiles
|
||||||
@@ -97,11 +113,12 @@ promptkit:
|
|||||||
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||||
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||||
|
|
||||||
Set at most one of **profile_dir** and **profile_file**. Relative values use
|
Set at most one of **profile_dir** and **profile_file**. Every nonempty relative
|
||||||
the process working directory, not the configuration file's directory. The
|
configured filesystem path is resolved from the canonical directory of the
|
||||||
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
|
file that supplies it. The complete example's
|
||||||
valid when Notarius is launched from the repository root; use an absolute path
|
`./profiles/dnd-extraction.yml` value therefore resolves beside that example.
|
||||||
for services and containers.
|
Use absolute paths for services and containers when their deployment layout is
|
||||||
|
not configuration-relative.
|
||||||
|
|
||||||
An operator source is optional. For a requested ID, PromptKit checks the
|
An operator source is optional. For a requested ID, PromptKit checks the
|
||||||
configured operator source first, then Notarius's embedded fallback profiles,
|
configured operator source first, then Notarius's embedded fallback profiles,
|
||||||
@@ -150,30 +167,15 @@ requiring credential values. It rejects absent, malformed, or incompatible
|
|||||||
profiles before a run prepares modules. Credential availability is checked only
|
profiles before a run prepares modules. Credential availability is checked only
|
||||||
when a generation is prepared.
|
when a generation is prepared.
|
||||||
|
|
||||||
## Migrating Version 3 Configuration
|
## Version 5 Migration
|
||||||
|
|
||||||
Version 3 files are not decoded or rewritten. Change **version: 3** to
|
Version 3 and version 4 files are rejected and are not rewritten at load time.
|
||||||
**version: 4** and rename the top-level **scriptorium:** section to
|
Use `version: 5`, rename the former `scriptorium` section to `promptkit`, and
|
||||||
**promptkit:**. Version 4 decoding is strict, so a remaining **scriptorium**
|
convert every module binding to an object containing `module`. Convert every
|
||||||
field is rejected as unknown.
|
external reference to `{path: ...}` and move any lane-level references to the
|
||||||
|
corresponding extraction binding. Put durable operational values in YAML or
|
||||||
## Operational Environment Variables
|
use the documented explicit run flags; the former field-level environment
|
||||||
|
variables are not read.
|
||||||
These variables are applied after YAML values:
|
|
||||||
|
|
||||||
| Variable | Overrides | Rules |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| **NOTARIUS_TOTAL_LLM_CONCURRENCY** | **concurrency.total_llm** | Integer. |
|
|
||||||
| **NOTARIUS_STAGE_WORKERS_EXTRACT** | **concurrency.stage_workers.extract** | Integer. |
|
|
||||||
| **NOTARIUS_OUTPUT_DIR** | **output.directory** | Non-empty path. |
|
|
||||||
| **NOTARIUS_CACHE_CHUNK_PLANS_MODE** | **cache.chunk_plans.mode** | **auto**, **bypass**, or **refresh**. |
|
|
||||||
| **NOTARIUS_CACHE_CHUNK_PLANS_DIR** | **cache.chunk_plans.directory** | Non-empty path. |
|
|
||||||
| **NOTARIUS_CACHE_CHECKPOINTS_DIR** | **cache.checkpoints.directory** | Non-empty path. |
|
|
||||||
| **NOTARIUS_DEBUG_DIR** | **debug.directory** | Non-empty path. |
|
|
||||||
|
|
||||||
Integer values are trimmed then parsed as base-10 integers. Directory and
|
|
||||||
output values reject NUL characters. **NOTARIUS_CONFIG** participates only in
|
|
||||||
configuration discovery.
|
|
||||||
|
|
||||||
## Concurrency, Output, Cache, And Debug
|
## Concurrency, Output, Cache, And Debug
|
||||||
|
|
||||||
@@ -198,13 +200,14 @@ debug:
|
|||||||
**concurrency.total_llm** must be greater than zero. The only supported
|
**concurrency.total_llm** must be greater than zero. The only supported
|
||||||
**concurrency.stage_workers** key is **extract**; its value must be from 1
|
**concurrency.stage_workers** key is **extract**; its value must be from 1
|
||||||
through **total_llm**. When omitted, it is recalculated from the effective
|
through **total_llm**. When omitted, it is recalculated from the effective
|
||||||
**total_llm** after YAML and environment precedence.
|
**total_llm** after imports, profile selection, and explicit ConfigKit
|
||||||
|
overrides.
|
||||||
|
|
||||||
**cache.chunk_plans.mode** accepts **auto**, **bypass**, or **refresh**.
|
**cache.chunk_plans.mode** accepts **auto**, **bypass**, or **refresh**.
|
||||||
**cache.checkpoints.enabled** is a boolean. The CLI can override the output
|
**cache.checkpoints.enabled** is a boolean. The CLI can override the output
|
||||||
directory and chunk-plan mode for one run; see [CLI reference](cli.md#run).
|
directory and chunk-plan mode for one run; see [CLI reference](cli.md#run).
|
||||||
Cache directory fields select only locations; Notarius has no cache permission
|
Cache directory fields select only locations; Notarius has no cache permission
|
||||||
field or environment override.
|
field or operational environment override.
|
||||||
|
|
||||||
## Pipelines
|
## Pipelines
|
||||||
|
|
||||||
@@ -214,14 +217,20 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
|
|||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
llm_profile: dnd-extraction
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input:
|
||||||
chunk: generic
|
module: seriatim
|
||||||
output: json
|
chunk:
|
||||||
|
module: generic
|
||||||
|
output:
|
||||||
|
module: json
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract: dnd/spells
|
extract:
|
||||||
merge: appendorder
|
module: dnd/spells
|
||||||
normalize: dnd/spells
|
merge:
|
||||||
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/spells
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
@@ -291,19 +300,20 @@ A lane has these fields:
|
|||||||
| **extract** | module binding | none | Required. |
|
| **extract** | module binding | none | Required. |
|
||||||
| **merge** | module binding | **appendorder** | Optional. |
|
| **merge** | module binding | **appendorder** | Optional. |
|
||||||
| **normalize** | module binding | **noop** | Optional. |
|
| **normalize** | module binding | **noop** | Optional. |
|
||||||
| **references** | map | none | Supported compatibility alias for **extract.references**. |
|
| **references** | map | none | Not supported at lane scope; place references on the eligible binding. |
|
||||||
| **validators** | list | none | Non-empty lane-level lists are rejected. Set validator overrides on a binding instead. |
|
| **validators** | list | none | Non-empty lane-level lists are rejected. Set validator overrides on a binding instead. |
|
||||||
|
|
||||||
The lane-level **references** alias remains accepted. When the alias and
|
Lane-level **references** are not supported. Put a lane extraction reference
|
||||||
**extract.references** bind the same slot, **extract.references** wins. Use
|
under **extract.references**.
|
||||||
the binding-local form in new configurations.
|
|
||||||
|
|
||||||
## Module Bindings And Validators
|
## Module Bindings And Validators
|
||||||
|
|
||||||
Use a module key directly when no other binding fields are needed:
|
Every module and validator binding uses an object, even when it has no fields
|
||||||
|
beyond its module key:
|
||||||
|
|
||||||
~~~yaml
|
~~~yaml
|
||||||
input: seriatim
|
input:
|
||||||
|
module: seriatim
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
Use an object for fields:
|
Use an object for fields:
|
||||||
@@ -314,7 +324,8 @@ extract:
|
|||||||
llm_profile: dnd-extraction
|
llm_profile: dnd-extraction
|
||||||
retries: 2
|
retries: 2
|
||||||
references:
|
references:
|
||||||
spell_catalog: ./dnd-spell-catalog.json
|
spell_catalog:
|
||||||
|
path: ./dnd-spell-catalog.json
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
| Binding field | Type | Default | Rules |
|
| Binding field | Type | Default | Rules |
|
||||||
@@ -386,18 +397,20 @@ When enabled, it publishes the selected source-unit excerpt defined by the
|
|||||||
|
|
||||||
## References And Ordered Handoffs
|
## References And Ordered Handoffs
|
||||||
|
|
||||||
Reference maps bind named slots that the selected target declares. A scalar is
|
Reference maps bind named slots that the selected target declares. An external
|
||||||
an external path. Pipeline-level maps accept only external paths; step-local
|
reference is an object with `path`. Pipeline-level maps accept only external
|
||||||
and binding-local maps may also select a normalized artifact from an earlier
|
paths; step-local and binding-local maps may also select a normalized artifact
|
||||||
step:
|
from an earlier step:
|
||||||
|
|
||||||
~~~yaml
|
~~~yaml
|
||||||
steps:
|
steps:
|
||||||
- id: describe-session
|
- id: describe-session
|
||||||
artifacts:
|
artifacts:
|
||||||
npc-registry:
|
npc-registry:
|
||||||
extract: dnd/npc-registry
|
extract:
|
||||||
normalize: dnd/npc-registry
|
module: dnd/npc-registry
|
||||||
|
normalize:
|
||||||
|
module: dnd/npc-registry
|
||||||
- id: extract-events
|
- id: extract-events
|
||||||
references:
|
references:
|
||||||
npc_registry:
|
npc_registry:
|
||||||
@@ -406,8 +419,10 @@ steps:
|
|||||||
lane: npc-registry
|
lane: npc-registry
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract: dnd/spells
|
extract:
|
||||||
normalize: dnd/spells
|
module: dnd/spells
|
||||||
|
normalize:
|
||||||
|
module: dnd/spells
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
An artifact selector contains only **step** and **lane**. The producer must be
|
An artifact selector contains only **step** and **lane**. The producer must be
|
||||||
@@ -422,9 +437,10 @@ bindings are then operational overrides of configured external paths; their
|
|||||||
pipeline, lane, and binding scopes and precedence are defined by the
|
pipeline, lane, and binding scopes and precedence are defined by the
|
||||||
[CLI reference](cli.md#reference-selectors). A CLI file reference cannot
|
[CLI reference](cli.md#reference-selectors). A CLI file reference cannot
|
||||||
replace a configured generated artifact handoff. Required slots must be bound
|
replace a configured generated artifact handoff. Required slots must be bound
|
||||||
after configuration and CLI reference actions are applied. Reference paths in
|
after configuration and CLI reference actions are applied. A YAML reference
|
||||||
YAML are resolved relative to the configuration file; CLI reference paths are
|
path resolves from the file that supplies its surviving declaration, including
|
||||||
resolved relative to the process working directory.
|
an imported fragment or selected profile. CLI reference paths resolve relative
|
||||||
|
to the process working directory.
|
||||||
|
|
||||||
### D&D Reference Slots
|
### D&D Reference Slots
|
||||||
|
|
||||||
@@ -573,7 +589,7 @@ go run ./cmd/notarius config validate \
|
|||||||
~~~
|
~~~
|
||||||
|
|
||||||
Configuration validation rejects invalid YAML, unsupported fields, invalid
|
Configuration validation rejects invalid YAML, unsupported fields, invalid
|
||||||
defaults or environment overrides, incompatible module keys, unknown options,
|
defaults or explicit CLI overrides, incompatible module keys, unknown options,
|
||||||
invalid reference bindings, missing required reference slots, invalid validator
|
invalid reference bindings, missing required reference slots, invalid validator
|
||||||
overrides, and incompatible generated artifact handoffs. Use
|
overrides, and incompatible generated artifact handoffs. Use
|
||||||
[pipelines list](cli.md#pipelines-list) to inspect configured IDs.
|
[pipelines list](cli.md#pipelines-list) to inspect configured IDs.
|
||||||
|
|||||||
@@ -23,16 +23,14 @@ A deployment must provide its own PromptKit profile and campaign reference
|
|||||||
files. Use absolute paths for service and subprocess deployments. In
|
files. Use absolute paths for service and subprocess deployments. In
|
||||||
particular, observe these different resolution rules:
|
particular, observe these different resolution rules:
|
||||||
|
|
||||||
- reference paths in YAML are resolved relative to the Notarius configuration
|
- reference and PromptKit profile paths in YAML are resolved relative to the
|
||||||
file;
|
file that supplies their surviving declaration;
|
||||||
- reference paths passed with `--reference` are resolved relative to the
|
- reference paths passed with `--reference` are resolved relative to the
|
||||||
Notarius process working directory; and
|
Notarius process working directory.
|
||||||
- `promptkit.profile_file` is resolved relative to the Notarius process working
|
|
||||||
directory.
|
|
||||||
|
|
||||||
Do not copy the repository example's relative profile path into a deployment
|
Do not copy the repository example's relative profile path into a deployment
|
||||||
without also controlling that working directory. The complete path and profile
|
without preserving its configuration-relative layout. The complete path and
|
||||||
rules are defined in [Configuration](../config.md).
|
profile rules are defined in [Configuration](../config.md).
|
||||||
|
|
||||||
Preflight the deployed configuration before processing sessions and whenever
|
Preflight the deployed configuration before processing sessions and whenever
|
||||||
it changes:
|
it changes:
|
||||||
|
|||||||
@@ -15,9 +15,12 @@ configured lane inventory, and downstream acceptance checklist, see
|
|||||||
Optionally preflight a selected configuration and pipeline before work starts:
|
Optionally preflight a selected configuration and pipeline before work starts:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
notarius config validate --config /path/to/notarius.yml --pipeline pipeline-id
|
notarius config validate --config /path/to/notarius.yml --config-profile production --pipeline pipeline-id
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Omit `--config-profile` when the composed base configuration is the intended
|
||||||
|
selection; it is independent from any `--llm-profile` run override.
|
||||||
|
|
||||||
Invoke the run with explicit paths and machine-readable output. Capture
|
Invoke the run with explicit paths and machine-readable output. Capture
|
||||||
standard output and standard error separately; do not combine them before
|
standard output and standard error separately; do not combine them before
|
||||||
processing the result.
|
processing the result.
|
||||||
@@ -25,6 +28,7 @@ processing the result.
|
|||||||
```sh
|
```sh
|
||||||
notarius run pipeline-id \
|
notarius run pipeline-id \
|
||||||
--config /path/to/notarius.yml \
|
--config /path/to/notarius.yml \
|
||||||
|
--config-profile production \
|
||||||
--input /path/to/source.json \
|
--input /path/to/source.json \
|
||||||
--output-dir /path/to/output-root \
|
--output-dir /path/to/output-root \
|
||||||
--json
|
--json
|
||||||
|
|||||||
@@ -126,11 +126,11 @@ notarius config validate --config /etc/notarius/config.yml --pipeline dnd-sessio
|
|||||||
An unset optional `api_key_env` reaches the provider without authorization and
|
An unset optional `api_key_env` reaches the provider without authorization and
|
||||||
may receive a 401 or 403 response.
|
may receive a 401 or 403 response.
|
||||||
|
|
||||||
Profile paths are currently resolved from the process working directory, not
|
Configured PromptKit profile paths resolve from the file that supplies their
|
||||||
from the configuration file. The complete example's
|
surviving declaration. The complete example's
|
||||||
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
|
`./profiles/dnd-extraction.yml` path is therefore relative to that example.
|
||||||
invocation only. Use absolute paths such as
|
Use absolute paths such as `/etc/notarius/profiles/dnd-extraction.yml` for
|
||||||
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
|
services and containers when that is clearer for the deployment.
|
||||||
|
|
||||||
## Run Lifecycle
|
## Run Lifecycle
|
||||||
|
|
||||||
@@ -308,7 +308,8 @@ Only a [debug-enabled run](cli.md#run) creates a bundle:
|
|||||||
trace/
|
trace/
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
The summary contains redacted invocation and resolution information plus run,
|
The summary contains redacted invocation and resolution information, including
|
||||||
|
`effective-config.yml` and the separately redacted resolved pipeline, plus run,
|
||||||
final grouped diagnostic, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
|
final grouped diagnostic, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
|
||||||
terminal records contain bounded attempt kinds, validator outcomes, policy,
|
terminal records contain bounded attempt kinds, validator outcomes, policy,
|
||||||
decision, PromptKit repair count, and usage; they do not contain assistant
|
decision, PromptKit repair count, and usage; they do not contain assistant
|
||||||
@@ -318,7 +319,8 @@ content. Neither surface is a cache input. Do not treat a debug bundle as safe
|
|||||||
to share merely because its configuration summary is redacted. Invocation
|
to share merely because its configuration summary is redacted. Invocation
|
||||||
metadata omits reasoning effort when it is inherited, records the replacement
|
metadata omits reasoning effort when it is inherited, records the replacement
|
||||||
value when one is supplied, and records an empty value when inherited reasoning
|
value when one is supplied, and records an empty value when inherited reasoning
|
||||||
was explicitly cleared.
|
was explicitly cleared. It records `config_profile` only when a non-base
|
||||||
|
configuration overlay was selected.
|
||||||
|
|
||||||
Notarius never creates debug state without an explicit request and never
|
Notarius never creates debug state without an explicit request and never
|
||||||
automatically deletes a requested bundle. If allocation succeeds, the command
|
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||||
|
|||||||
@@ -484,6 +484,8 @@ incompatible generated references.
|
|||||||
|
|
||||||
## Stage 10 — Switch Run And The Maintained V5 Configurations
|
## Stage 10 — Switch Run And The Maintained V5 Configurations
|
||||||
|
|
||||||
|
**Status:** Completed 2026-09-13.
|
||||||
|
|
||||||
**Goal:** Complete production cutover and make all runs use one ConfigKit load.
|
**Goal:** Complete production cutover and make all runs use one ConfigKit load.
|
||||||
|
|
||||||
**Work:**
|
**Work:**
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
version: 4
|
version: 5
|
||||||
promptkit:
|
promptkit:
|
||||||
profile_file: ./examples/profiles/dnd-extraction.yml
|
profile_file: ./profiles/dnd-extraction.yml
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -19,11 +19,14 @@ debug:
|
|||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
llm_profile: dnd-extraction
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input:
|
||||||
|
module: seriatim
|
||||||
# Stable campaign context is shared by every module that accepts these slots.
|
# Stable campaign context is shared by every module that accepts these slots.
|
||||||
references:
|
references:
|
||||||
party: ./dnd-party.txt
|
party:
|
||||||
glossary: ./dnd-glossary.txt
|
path: ./dnd-party.txt
|
||||||
|
glossary:
|
||||||
|
path: ./dnd-glossary.txt
|
||||||
chunk:
|
chunk:
|
||||||
module: dnd/scenes
|
module: dnd/scenes
|
||||||
retries: 2
|
retries: 2
|
||||||
@@ -52,13 +55,16 @@ pipelines:
|
|||||||
extract:
|
extract:
|
||||||
module: dnd/item-registry
|
module: dnd/item-registry
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
normalize: dnd/item-registry
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/item-registry
|
||||||
npc-registry:
|
npc-registry:
|
||||||
extract:
|
extract:
|
||||||
module: dnd/npc-registry
|
module: dnd/npc-registry
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
|
module: appendorder
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/npc-registry
|
module: dnd/npc-registry
|
||||||
retries: 2
|
retries: 2
|
||||||
@@ -66,7 +72,8 @@ pipelines:
|
|||||||
extract:
|
extract:
|
||||||
module: dnd/location-registry
|
module: dnd/location-registry
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
|
module: appendorder
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/location-registry
|
module: dnd/location-registry
|
||||||
retries: 2
|
retries: 2
|
||||||
@@ -74,8 +81,10 @@ pipelines:
|
|||||||
extract:
|
extract:
|
||||||
module: dnd/scene-descriptions
|
module: dnd/scene-descriptions
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
normalize: dnd/scene-descriptions
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/scene-descriptions
|
||||||
- id: extract-events
|
- id: extract-events
|
||||||
# Accepted registry artifacts and scene-description eligibility artifacts
|
# Accepted registry artifacts and scene-description eligibility artifacts
|
||||||
# are supplied in memory to their compatible consumers in this step.
|
# are supplied in memory to their compatible consumers in this step.
|
||||||
@@ -101,38 +110,49 @@ pipelines:
|
|||||||
extract:
|
extract:
|
||||||
module: dnd/item-occurrences
|
module: dnd/item-occurrences
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
normalize: dnd/item-occurrences
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/item-occurrences
|
||||||
spells:
|
spells:
|
||||||
extract:
|
extract:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
retries: 2
|
retries: 2
|
||||||
references:
|
references:
|
||||||
spell_catalog: ./dnd-spell-catalog.json
|
spell_catalog:
|
||||||
merge: appendorder
|
path: ./dnd-spell-catalog.json
|
||||||
|
merge:
|
||||||
|
module: appendorder
|
||||||
# Stage-local file references are intentionally bound at each stage.
|
# Stage-local file references are intentionally bound at each stage.
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
references:
|
references:
|
||||||
spell_catalog: ./dnd-spell-catalog.json
|
spell_catalog:
|
||||||
|
path: ./dnd-spell-catalog.json
|
||||||
combat-turns:
|
combat-turns:
|
||||||
extract:
|
extract:
|
||||||
module: dnd/combat-turns
|
module: dnd/combat-turns
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
normalize: dnd/combat-turns
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/combat-turns
|
||||||
npc-occurrences:
|
npc-occurrences:
|
||||||
extract:
|
extract:
|
||||||
module: dnd/npc-occurrences
|
module: dnd/npc-occurrences
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
normalize: dnd/npc-occurrences
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/npc-occurrences
|
||||||
location-occurrences:
|
location-occurrences:
|
||||||
extract:
|
extract:
|
||||||
module: dnd/location-occurrences
|
module: dnd/location-occurrences
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
normalize: dnd/location-occurrences
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/location-occurrences
|
||||||
- id: track-enemies
|
- id: track-enemies
|
||||||
references:
|
references:
|
||||||
npc_registry:
|
npc_registry:
|
||||||
@@ -156,5 +176,7 @@ pipelines:
|
|||||||
extract:
|
extract:
|
||||||
module: dnd/enemy-events
|
module: dnd/enemy-events
|
||||||
retries: 2
|
retries: 2
|
||||||
merge: appendorder
|
merge:
|
||||||
normalize: dnd/enemy-events
|
module: appendorder
|
||||||
|
normalize:
|
||||||
|
module: dnd/enemy-events
|
||||||
|
|||||||
@@ -1,9 +1,12 @@
|
|||||||
version: 4
|
version: 5
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
llm_profile: dnd-extraction
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input:
|
||||||
|
module: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract: dnd/spells
|
extract:
|
||||||
normalize: dnd/spells
|
module: dnd/spells
|
||||||
|
normalize:
|
||||||
|
module: dnd/spells
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ func TestRunChunkPlanModePrecedenceAndValidation(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{name: "default", wantStores: 1},
|
{name: "default", wantStores: 1},
|
||||||
{name: "file", fileMode: "bypass"},
|
{name: "file", fileMode: "bypass"},
|
||||||
{name: "environment", envMode: "bypass"},
|
{name: "retired environment has no effect", envMode: "bypass", wantStores: 1},
|
||||||
{name: "cli", envMode: "refresh", cliMode: "bypass"},
|
{name: "cli", envMode: "refresh", cliMode: "bypass"},
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
@@ -81,24 +81,13 @@ func TestRunChunkPlanModePrecedenceAndValidation(t *testing.T) {
|
|||||||
|
|
||||||
for _, tt := range []struct {
|
for _, tt := range []struct {
|
||||||
name string
|
name string
|
||||||
fileConfig bool
|
|
||||||
}{
|
}{
|
||||||
{name: "invalid environment mode"},
|
{name: "invalid file mode"},
|
||||||
{name: "invalid file mode", fileConfig: true},
|
|
||||||
} {
|
} {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
if tt.fileConfig {
|
|
||||||
replaceStateTestConfigLine(t, roots.config, " mode: auto\n", " mode: invalid\n")
|
replaceStateTestConfigLine(t, roots.config, " mode: auto\n", " mode: invalid\n")
|
||||||
} else {
|
|
||||||
opts.LookupEnv = func(name string) (string, bool) {
|
|
||||||
if name == "NOTARIUS_CACHE_CHUNK_PLANS_MODE" {
|
|
||||||
return "invalid", true
|
|
||||||
}
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts)
|
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts)
|
||||||
if code != 1 || stdout.Len() != 0 || stderr.Len() == 0 {
|
if code != 1 || stdout.Len() != 0 || stderr.Len() == 0 {
|
||||||
@@ -207,16 +196,20 @@ func TestRunAutoReusesPlanWhenRunInputsChange(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
configText := replaceRequiredOnce(t, string(data), " chunk: test/chunk\n", ` chunk:
|
configText := replaceRequiredOnce(t, string(data), " chunk:\n module: test/chunk\n", ` chunk:
|
||||||
module: test/chunk
|
module: test/chunk
|
||||||
options:
|
options:
|
||||||
strategy: first
|
strategy: first
|
||||||
`)
|
`)
|
||||||
configText = replaceRequiredOnce(t, configText, " output: test/output\n", ` other:
|
configText = replaceRequiredOnce(t, configText, " output:\n module: test/output\n", ` other:
|
||||||
extract: test/extract
|
extract:
|
||||||
merge: test/merge
|
module: test/extract
|
||||||
normalize: test/normalize
|
merge:
|
||||||
output: test/output
|
module: test/merge
|
||||||
|
normalize:
|
||||||
|
module: test/normalize
|
||||||
|
output:
|
||||||
|
module: test/output
|
||||||
`)
|
`)
|
||||||
if err := os.WriteFile(roots.config, []byte(configText), 0o600); err != nil {
|
if err := os.WriteFile(roots.config, []byte(configText), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -140,7 +140,7 @@ func TestConfigLoadingFailuresReturnOneWithPathContext(t *testing.T) {
|
|||||||
stdout.Reset()
|
stdout.Reset()
|
||||||
stderr.Reset()
|
stderr.Reset()
|
||||||
code = RunWithOptions([]string{"config", "validate", "--config", malformed}, &stdout, &stderr, commandContractOptions(t))
|
code = RunWithOptions([]string{"config", "validate", "--config", malformed}, &stdout, &stderr, commandContractOptions(t))
|
||||||
if code != 1 || !strings.Contains(stderr.String(), malformed) || !strings.Contains(stderr.String(), "parse config file") || stdout.Len() != 0 {
|
if code != 1 || !strings.Contains(stderr.String(), "configuration contains invalid YAML") || stdout.Len() != 0 {
|
||||||
t.Fatalf("malformed config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
t.Fatalf("malformed config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -185,14 +185,16 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
|
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
|
||||||
configPath := writeCommandConfigContent(t, `version: 4
|
configPath := writeCommandConfigContent(t, `version: 5
|
||||||
pipelines:
|
pipelines:
|
||||||
demo:
|
demo:
|
||||||
llm_profile: dnd-extraction
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input:
|
||||||
|
module: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract: dnd/spells
|
extract:
|
||||||
|
module: dnd/spells
|
||||||
`)
|
`)
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
|
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
|
||||||
@@ -261,19 +263,21 @@ func commandContractOptionsWithLookup(t *testing.T, lookup func(string) (string,
|
|||||||
|
|
||||||
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
content := fmt.Sprintf("version: 4\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
|
content := fmt.Sprintf("version: 5\npipelines:\n %q:\n input:\n module: seriatim\n %q:\n input:\n module: seriatim\n", firstID, secondID)
|
||||||
return writeCommandConfigContent(t, content)
|
return writeCommandConfigContent(t, content)
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeResolvableCommandConfig(t *testing.T) string {
|
func writeResolvableCommandConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return writeCommandConfigContent(t, `version: 4
|
return writeCommandConfigContent(t, `version: 5
|
||||||
pipelines:
|
pipelines:
|
||||||
demo:
|
demo:
|
||||||
input: seriatim
|
input:
|
||||||
|
module: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract: dnd/spells
|
extract:
|
||||||
|
module: dnd/spells
|
||||||
`)
|
`)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
func runConfigKitValidate(args []string, stdout, stderr io.Writer, opts Options) int {
|
func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||||
fs := flag.NewFlagSet("config validate", flag.ContinueOnError)
|
fs := flag.NewFlagSet("config validate", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
configPath := fs.String("config", "", "config file path")
|
configPath := fs.String("config", "", "config file path")
|
||||||
@@ -62,7 +62,7 @@ func runConfigKitValidate(args []string, stdout, stderr io.Writer, opts Options)
|
|||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
func runConfigKitPipelinesList(args []string, stdout, stderr io.Writer, opts Options) int {
|
func runPipelinesList(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||||
fs := flag.NewFlagSet("pipelines list", flag.ContinueOnError)
|
fs := flag.NewFlagSet("pipelines list", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
configPath := fs.String("config", "", "config file path")
|
configPath := fs.String("config", "", "config file path")
|
||||||
|
|||||||
@@ -12,14 +12,15 @@ func TestConfigKitReadOnlyCommandsLoadImportsAndProfiles(t *testing.T) {
|
|||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
path := filepath.Join(root, "notarius.yml")
|
path := filepath.Join(root, "notarius.yml")
|
||||||
writeConfigKitFile(t, filepath.Join(root, "fragment.yml"), "pipelines:\n imported:\n input: {module: input}\n")
|
writeConfigKitFile(t, filepath.Join(root, "fragment.yml"), "pipelines:\n imported:\n input: {module: input}\n")
|
||||||
|
writeConfigKitFile(t, filepath.Join(root, "profiles", "development.yml"), "pipelines:\n profiled:\n input: {module: input}\n")
|
||||||
writeConfigKitFile(t, path, "version: 5\nconfigkit:\n imports:\n - path: fragment.yml\npipelines:\n base:\n input: {module: input}\n")
|
writeConfigKitFile(t, path, "version: 5\nconfigkit:\n imports:\n - path: fragment.yml\npipelines:\n base:\n input: {module: input}\n")
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
if code := runConfigKitPipelinesList([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 0 || stdout.String() != "base\nimported\n" {
|
if code := RunWithOptions([]string{"pipelines", "list", "--config", path, "--config-profile", "development"}, &stdout, &stderr, commandContractOptions(t)); code != 0 || stdout.String() != "base\nimported\nprofiled\n" {
|
||||||
t.Fatalf("list = %d, %q, %q", code, stdout.String(), stderr.String())
|
t.Fatalf("list = %d, %q, %q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
stdout.Reset()
|
stdout.Reset()
|
||||||
stderr.Reset()
|
stderr.Reset()
|
||||||
if code := runConfigKitValidate([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 0 || !strings.Contains(stdout.String(), "is valid") {
|
if code := RunWithOptions([]string{"config", "validate", "--config", path, "--config-profile", "development"}, &stdout, &stderr, commandContractOptions(t)); code != 0 || !strings.Contains(stdout.String(), "is valid") {
|
||||||
t.Fatalf("validate = %d, %q, %q", code, stdout.String(), stderr.String())
|
t.Fatalf("validate = %d, %q, %q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -29,7 +30,7 @@ func TestConfigKitReadOnlyCommandsRejectVersionFour(t *testing.T) {
|
|||||||
path := filepath.Join(t.TempDir(), "notarius.yml")
|
path := filepath.Join(t.TempDir(), "notarius.yml")
|
||||||
writeConfigKitFile(t, path, "version: 4\n")
|
writeConfigKitFile(t, path, "version: 4\n")
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
if code := runConfigKitPipelinesList([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 1 || !strings.Contains(stderr.String(), "version") {
|
if code := runPipelinesList([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 1 || !strings.Contains(stderr.String(), "version") {
|
||||||
t.Fatalf("list = %d, %q", code, stderr.String())
|
t.Fatalf("list = %d, %q", code, stderr.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -267,6 +267,7 @@ func completeExampleConfigWithTemporaryCache(t *testing.T) string {
|
|||||||
"./dnd-party.txt": repositoryPath("examples", "dnd-party.txt"),
|
"./dnd-party.txt": repositoryPath("examples", "dnd-party.txt"),
|
||||||
"./dnd-glossary.txt": repositoryPath("examples", "dnd-glossary.txt"),
|
"./dnd-glossary.txt": repositoryPath("examples", "dnd-glossary.txt"),
|
||||||
"./dnd-spell-catalog.json": repositoryPath("examples", "dnd-spell-catalog.json"),
|
"./dnd-spell-catalog.json": repositoryPath("examples", "dnd-spell-catalog.json"),
|
||||||
|
"./profiles/dnd-extraction.yml": repositoryPath("examples", "profiles", "dnd-extraction.yml"),
|
||||||
} {
|
} {
|
||||||
updated = strings.ReplaceAll(updated, relative, fmt.Sprintf("%q", absolute))
|
updated = strings.ReplaceAll(updated, relative, fmt.Sprintf("%q", absolute))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
||||||
content := fmt.Sprintf(`version: 4
|
content := fmt.Sprintf(`version: 5
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
@@ -32,14 +32,17 @@ cache:
|
|||||||
directory: %q
|
directory: %q
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
input: seriatim
|
input:
|
||||||
|
module: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract:
|
extract:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
references:
|
references:
|
||||||
npc_registry: %q
|
npc_registry:
|
||||||
normalize: dnd/spells
|
path: %q
|
||||||
|
normalize:
|
||||||
|
module: dnd/spells
|
||||||
`, checkpointRoot, npcPath)
|
`, checkpointRoot, npcPath)
|
||||||
configPath := filepath.Join(t.TempDir(), "config.yml")
|
configPath := filepath.Join(t.TempDir(), "config.yml")
|
||||||
if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil {
|
if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil {
|
||||||
|
|||||||
@@ -737,13 +737,13 @@ func TestProductionConfigValidationCoversModuleAndVariantFailures(t *testing.T)
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "unknown module",
|
name: "unknown module",
|
||||||
content: replaceRequiredOnce(t, base, " input: seriatim\n", " input: missing/input\n"),
|
content: replaceRequiredOnce(t, base, " input:\n module: seriatim\n", " input:\n module: missing/input\n"),
|
||||||
options: productionCLIOptions(t),
|
options: productionCLIOptions(t),
|
||||||
fragments: []string{"pipeline \"dnd-session\"", "input", "missing/input"},
|
fragments: []string{"pipeline \"dnd-session\"", "input", "missing/input"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "unknown validator",
|
name: "unknown validator",
|
||||||
content: replaceRequiredOnce(t, base, " extract: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: missing/validator\n"),
|
content: replaceRequiredOnce(t, base, " extract:\n module: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: missing/validator\n"),
|
||||||
options: productionCLIOptions(t),
|
options: productionCLIOptions(t),
|
||||||
fragments: []string{"validator", "missing/validator"},
|
fragments: []string{"validator", "missing/validator"},
|
||||||
},
|
},
|
||||||
@@ -755,7 +755,7 @@ func TestProductionConfigValidationCoversModuleAndVariantFailures(t *testing.T)
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "deterministic validator with profile",
|
name: "deterministic validator with profile",
|
||||||
content: replaceRequiredOnce(t, base, " extract: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: generic/valid_json\n llm_profile: forbidden-profile\n"),
|
content: replaceRequiredOnce(t, base, " extract:\n module: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: generic/valid_json\n llm_profile: forbidden-profile\n"),
|
||||||
options: productionCLIOptions(t),
|
options: productionCLIOptions(t),
|
||||||
fragments: []string{"deterministic validator", "llm_profile"},
|
fragments: []string{"deterministic validator", "llm_profile"},
|
||||||
},
|
},
|
||||||
@@ -779,7 +779,7 @@ func TestProductionConfigValidationCoversModuleAndVariantFailures(t *testing.T)
|
|||||||
|
|
||||||
func TestProductionNormalizeValidatorOverrideRemainsAuthoritative(t *testing.T) {
|
func TestProductionNormalizeValidatorOverrideRemainsAuthoritative(t *testing.T) {
|
||||||
base := string(readRepositoryFile(t, "examples", "dnd-minimal.config.yml"))
|
base := string(readRepositoryFile(t, "examples", "dnd-minimal.config.yml"))
|
||||||
content := replaceRequiredOnce(t, base, " normalize: dnd/spells\n", " normalize:\n module: dnd/spells\n validators:\n - module: generic/always_accept\n - module: generic/valid_json\n")
|
content := replaceRequiredOnce(t, base, " normalize:\n module: dnd/spells\n", " normalize:\n module: dnd/spells\n validators:\n - module: generic/always_accept\n - module: generic/valid_json\n")
|
||||||
path := writeProductionContractConfig(t, content)
|
path := writeProductionContractConfig(t, content)
|
||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
effective, err := loadMaintainedExample(t, path).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
|
effective, err := loadMaintainedExample(t, path).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
|
||||||
@@ -879,7 +879,7 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
|||||||
|
|
||||||
func productionSpellCatalogContractConfig(t *testing.T) string {
|
func productionSpellCatalogContractConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return fmt.Sprintf(`version: 4
|
return fmt.Sprintf(`version: 5
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
@@ -888,21 +888,26 @@ cache:
|
|||||||
directory: ""
|
directory: ""
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
input: seriatim
|
input:
|
||||||
|
module: seriatim
|
||||||
references:
|
references:
|
||||||
party: %q
|
party:
|
||||||
glossary: %q
|
path: %q
|
||||||
|
glossary:
|
||||||
|
path: %q
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract:
|
extract:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
retries: 2
|
retries: 2
|
||||||
references:
|
references:
|
||||||
spell_catalog: %q
|
spell_catalog:
|
||||||
|
path: %q
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
references:
|
references:
|
||||||
spell_catalog: %q
|
spell_catalog:
|
||||||
|
path: %q
|
||||||
`, repositoryPath("examples", "dnd-party.txt"), repositoryPath("examples", "dnd-glossary.txt"), repositoryPath("examples", "dnd-spell-catalog.json"), repositoryPath("examples", "dnd-spell-catalog.json"))
|
`, repositoryPath("examples", "dnd-party.txt"), repositoryPath("examples", "dnd-glossary.txt"), repositoryPath("examples", "dnd-spell-catalog.json"), repositoryPath("examples", "dnd-spell-catalog.json"))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -913,14 +918,11 @@ func writeProductionSpellCatalogContractConfig(t *testing.T) string {
|
|||||||
|
|
||||||
func loadMaintainedExample(t *testing.T, path string) config.Config {
|
func loadMaintainedExample(t *testing.T, path string) config.Config {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
fileConfig, err := config.LoadFileConfig(path)
|
loaded, err := config.Load(path, config.LoadOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("load maintained config %q: %v", path, err)
|
t.Fatalf("load maintained config %q: %v", path, err)
|
||||||
}
|
}
|
||||||
cfg := config.Default()
|
cfg := loaded.RuntimeConfig()
|
||||||
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
|
|
||||||
t.Fatalf("apply maintained config %q: %v", path, err)
|
|
||||||
}
|
|
||||||
if err := cfg.Validate(); err != nil {
|
if err := cfg.Validate(); err != nil {
|
||||||
t.Fatalf("validate maintained config %q: %v", path, err)
|
t.Fatalf("validate maintained config %q: %v", path, err)
|
||||||
}
|
}
|
||||||
@@ -977,7 +979,7 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func productionRunConfig(outputRoot, chunkModule string) string {
|
func productionRunConfig(outputRoot, chunkModule string) string {
|
||||||
return fmt.Sprintf(`version: 4
|
return fmt.Sprintf(`version: 5
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
@@ -988,8 +990,10 @@ debug:
|
|||||||
directory: %q
|
directory: %q
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
input: seriatim
|
input:
|
||||||
chunk: %s
|
module: seriatim
|
||||||
|
chunk:
|
||||||
|
module: %s
|
||||||
output:
|
output:
|
||||||
module: json
|
module: json
|
||||||
options:
|
options:
|
||||||
@@ -998,8 +1002,10 @@ pipelines:
|
|||||||
- id: identify-items
|
- id: identify-items
|
||||||
artifacts:
|
artifacts:
|
||||||
item-registry:
|
item-registry:
|
||||||
extract: dnd/item-registry
|
extract:
|
||||||
normalize: dnd/item-registry
|
module: dnd/item-registry
|
||||||
|
normalize:
|
||||||
|
module: dnd/item-registry
|
||||||
- id: extract-occurrences
|
- id: extract-occurrences
|
||||||
references:
|
references:
|
||||||
item_registry:
|
item_registry:
|
||||||
@@ -1008,9 +1014,11 @@ pipelines:
|
|||||||
lane: item-registry
|
lane: item-registry
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
extract: dnd/spells
|
extract:
|
||||||
|
module: dnd/spells
|
||||||
item-occurrences:
|
item-occurrences:
|
||||||
extract: dnd/item-occurrences
|
extract:
|
||||||
|
module: dnd/item-occurrences
|
||||||
`, outputRoot, filepath.Join(filepath.Dir(outputRoot), "debug"), chunkModule)
|
`, outputRoot, filepath.Join(filepath.Dir(outputRoot), "debug"), chunkModule)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -194,7 +194,7 @@ func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest
|
|||||||
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
config := fmt.Sprintf(`version: 4
|
config := fmt.Sprintf(`version: 5
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
@@ -208,19 +208,27 @@ debug:
|
|||||||
directory: %q
|
directory: %q
|
||||||
pipelines:
|
pipelines:
|
||||||
sample:
|
sample:
|
||||||
input: test/input
|
input:
|
||||||
chunk: test/chunk
|
module: test/input
|
||||||
|
chunk:
|
||||||
|
module: test/chunk
|
||||||
steps:
|
steps:
|
||||||
- id: first
|
- id: first
|
||||||
artifacts:
|
artifacts:
|
||||||
producer:
|
producer:
|
||||||
extract: test/extract/producer
|
extract:
|
||||||
merge: test/merge
|
module: test/extract/producer
|
||||||
normalize: test/normalize
|
merge:
|
||||||
|
module: test/merge
|
||||||
|
normalize:
|
||||||
|
module: test/normalize
|
||||||
unrelated:
|
unrelated:
|
||||||
extract: test/extract/unrelated
|
extract:
|
||||||
merge: test/merge
|
module: test/extract/unrelated
|
||||||
normalize: test/normalize
|
merge:
|
||||||
|
module: test/merge
|
||||||
|
normalize:
|
||||||
|
module: test/normalize
|
||||||
- id: second
|
- id: second
|
||||||
references:
|
references:
|
||||||
upstream:
|
upstream:
|
||||||
@@ -229,9 +237,12 @@ pipelines:
|
|||||||
lane: producer
|
lane: producer
|
||||||
artifacts:
|
artifacts:
|
||||||
middle:
|
middle:
|
||||||
extract: test/extract/middle
|
extract:
|
||||||
merge: test/merge
|
module: test/extract/middle
|
||||||
normalize: test/normalize
|
merge:
|
||||||
|
module: test/merge
|
||||||
|
normalize:
|
||||||
|
module: test/normalize
|
||||||
- id: third
|
- id: third
|
||||||
references:
|
references:
|
||||||
upstream:
|
upstream:
|
||||||
@@ -240,10 +251,14 @@ pipelines:
|
|||||||
lane: middle
|
lane: middle
|
||||||
artifacts:
|
artifacts:
|
||||||
dependent:
|
dependent:
|
||||||
extract: test/extract/dependent
|
extract:
|
||||||
merge: test/merge
|
module: test/extract/dependent
|
||||||
normalize: test/normalize
|
merge:
|
||||||
output: test/recompute-output
|
module: test/merge
|
||||||
|
normalize:
|
||||||
|
module: test/normalize
|
||||||
|
output:
|
||||||
|
module: test/recompute-output
|
||||||
`, roots.output, roots.plans, roots.checkpoints, roots.debug)
|
`, roots.output, roots.plans, roots.checkpoints, roots.debug)
|
||||||
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -289,7 +289,7 @@ func TestReferenceMaterializationSeparatesCLIAndConfigPathOrigins(t *testing.T)
|
|||||||
workingDir := t.TempDir()
|
workingDir := t.TempDir()
|
||||||
cfg := referenceContractConfig()
|
cfg := referenceContractConfig()
|
||||||
configPath := filepath.Join(configDir, "config.yml")
|
configPath := filepath.Join(configDir, "config.yml")
|
||||||
if err := os.WriteFile(configPath, []byte("version: 4\n"), 0o600); err != nil {
|
if err := os.WriteFile(configPath, []byte("version: 5\n"), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"encoding/hex"
|
"encoding/hex"
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
"errors"
|
||||||
"flag"
|
"flag"
|
||||||
"fmt"
|
"fmt"
|
||||||
@@ -16,6 +15,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/buildinfo"
|
"gitea.maximumdirect.net/eric/notarius/internal/buildinfo"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
@@ -34,8 +34,8 @@ const usage = `Usage:
|
|||||||
notarius help
|
notarius help
|
||||||
notarius --version
|
notarius --version
|
||||||
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
|
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
|
||||||
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
|
notarius config validate --config path/to/config.yml [--config-profile name] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
notarius pipelines list --config path/to/config.yml [--json]
|
notarius pipelines list --config path/to/config.yml [--config-profile name] [--json]
|
||||||
`
|
`
|
||||||
|
|
||||||
type Options struct {
|
type Options struct {
|
||||||
@@ -158,6 +158,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
fs := flag.NewFlagSet("run", flag.ContinueOnError)
|
fs := flag.NewFlagSet("run", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
configPath := fs.String("config", "", "config file path")
|
configPath := fs.String("config", "", "config file path")
|
||||||
|
configProfile := fs.String("config-profile", "", "configuration profile")
|
||||||
inputPath := fs.String("input", "", "source input file path")
|
inputPath := fs.String("input", "", "source input file path")
|
||||||
onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
|
onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
|
||||||
outputDir := fs.String("output-dir", "", "output directory")
|
outputDir := fs.String("output-dir", "", "output directory")
|
||||||
@@ -265,20 +266,12 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
return 2
|
return 2
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg, loadedConfigPath, err := loadConfig(*configPath, opts)
|
loaded, loadedConfigPath, err := loadConfigKit(*configPath, *configProfile, runConfigurationOverrides(chunkCache, *outputDir, *debugDir), opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
writeConfigLoadError(stderr, err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
if chunkCache.set {
|
cfg := loaded.RuntimeConfig()
|
||||||
cfg.Cache.ChunkPlans.Mode = chunkCache.value
|
|
||||||
}
|
|
||||||
if dir := strings.TrimSpace(*outputDir); dir != "" {
|
|
||||||
cfg.Output.Directory = dir
|
|
||||||
}
|
|
||||||
if dir := strings.TrimSpace(*debugDir); dir != "" {
|
|
||||||
cfg.Debug.Directory = dir
|
|
||||||
}
|
|
||||||
if err := cfg.Validate(); err != nil {
|
if err := cfg.Validate(); err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
@@ -336,6 +329,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
InputPath: strings.TrimSpace(*inputPath),
|
InputPath: strings.TrimSpace(*inputPath),
|
||||||
ConfigPath: loadedConfigPath,
|
ConfigPath: loadedConfigPath,
|
||||||
ConfigSource: configSource(*configPath),
|
ConfigSource: configSource(*configPath),
|
||||||
|
ConfigProfile: effectiveConfigProfile(*configProfile),
|
||||||
OnlyLanes: append([]string(nil), only...),
|
OnlyLanes: append([]string(nil), only...),
|
||||||
ChunkCacheOverride: chunkCache.explicitValue(),
|
ChunkCacheOverride: chunkCache.explicitValue(),
|
||||||
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
|
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
|
||||||
@@ -394,7 +388,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
|
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
|
||||||
}
|
}
|
||||||
if err := writeSummary(summary, func() error { return summary.WriteRedactedEffectiveConfig(effective) }); err != nil {
|
if err := writeSummary(summary, func() error { return summary.WriteEffectiveConfig(loaded) }); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug effective config: %w", err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug effective config: %w", err))
|
||||||
}
|
}
|
||||||
if err := writeSummary(summary, func() error { return summary.WriteResolvedPipeline(effective) }); err != nil {
|
if err := writeSummary(summary, func() error { return summary.WriteResolvedPipeline(effective) }); err != nil {
|
||||||
@@ -881,13 +875,39 @@ func reorderRunArgs(args []string) []string {
|
|||||||
|
|
||||||
func runFlagTakesValue(arg string) bool {
|
func runFlagTakesValue(arg string) bool {
|
||||||
switch arg {
|
switch arg {
|
||||||
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
case "--config", "--config-profile", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
||||||
return true
|
return true
|
||||||
default:
|
default:
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func runConfigurationOverrides(chunkCache chunkCacheFlag, outputDir, debugDir string) []configkit.Override {
|
||||||
|
overrides := make([]configkit.Override, 0, 3)
|
||||||
|
if chunkCache.set {
|
||||||
|
overrides = append(overrides, configkit.Override{
|
||||||
|
Path: configkit.Keys("cache", "chunk_plans", "mode"),
|
||||||
|
Value: string(chunkCache.value),
|
||||||
|
Flag: "--chunk_cache",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if outputDir = strings.TrimSpace(outputDir); outputDir != "" {
|
||||||
|
overrides = append(overrides, configkit.Override{
|
||||||
|
Path: configkit.Keys("output", "directory"),
|
||||||
|
Value: outputDir,
|
||||||
|
Flag: "--output-dir",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if debugDir = strings.TrimSpace(debugDir); debugDir != "" {
|
||||||
|
overrides = append(overrides, configkit.Override{
|
||||||
|
Path: configkit.Keys("debug", "directory"),
|
||||||
|
Value: debugDir,
|
||||||
|
Flag: "--debug-dir",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return overrides
|
||||||
|
}
|
||||||
|
|
||||||
type chunkCacheFlag struct {
|
type chunkCacheFlag struct {
|
||||||
value pipeline.ChunkCacheMode
|
value pipeline.ChunkCacheMode
|
||||||
set bool
|
set bool
|
||||||
@@ -941,7 +961,7 @@ func chunkPlanStoreForRun(cfg config.ChunkPlanCacheConfig, opts Options) (pipeli
|
|||||||
|
|
||||||
func validateRunFlagValues(args []string) error {
|
func validateRunFlagValues(args []string) error {
|
||||||
for i, arg := range args {
|
for i, arg := range args {
|
||||||
if arg != "--session-id" && arg != "--reasoning-effort" {
|
if arg != "--config-profile" && arg != "--session-id" && arg != "--reasoning-effort" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
|
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
|
||||||
@@ -951,6 +971,13 @@ func validateRunFlagValues(args []string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func effectiveConfigProfile(profile string) string {
|
||||||
|
if profile == "" || profile == "default" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return profile
|
||||||
|
}
|
||||||
|
|
||||||
func flagWasProvided(args []string, name string) bool {
|
func flagWasProvided(args []string, name string) bool {
|
||||||
for _, arg := range args {
|
for _, arg := range args {
|
||||||
if arg == name || strings.HasPrefix(arg, name+"=") {
|
if arg == name || strings.HasPrefix(arg, name+"=") {
|
||||||
@@ -1033,67 +1060,6 @@ func runConfig(args []string, stdout, stderr io.Writer, opts Options) int {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) int {
|
|
||||||
fs := flag.NewFlagSet("config validate", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
configPath := fs.String("config", "", "config file path")
|
|
||||||
pipelineID := fs.String("pipeline", "", "pipeline ID")
|
|
||||||
onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
fmt.Fprintf(stderr, "notarius: unexpected argument %q\n", fs.Arg(0))
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(*onlyRaw) != "" && strings.TrimSpace(*pipelineID) == "" {
|
|
||||||
fmt.Fprintln(stderr, "notarius: --only requires --pipeline")
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
only, err := parseOnly(*onlyRaw)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg, path, err := loadConfig(*configPath, opts)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.TrimSpace(*pipelineID) != "" {
|
|
||||||
catalog, err := effectiveCatalog(opts)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
effective, err := cfg.Resolve(config.ResolveInput{
|
|
||||||
PipelineID: *pipelineID,
|
|
||||||
Only: only,
|
|
||||||
Catalog: catalog,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
fmt.Fprintf(stdout, "config %q is valid for pipeline %q\n", path, strings.TrimSpace(*pipelineID))
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := cfg.Validate(); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
fmt.Fprintf(stdout, "config %q is valid\n", path)
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
func runPipelines(args []string, stdout, stderr io.Writer, opts Options) int {
|
func runPipelines(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||||
if len(args) == 0 {
|
if len(args) == 0 {
|
||||||
fmt.Fprintln(stderr, "notarius: pipelines requires a subcommand")
|
fmt.Fprintln(stderr, "notarius: pipelines requires a subcommand")
|
||||||
@@ -1110,50 +1076,6 @@ func runPipelines(args []string, stdout, stderr io.Writer, opts Options) int {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func runPipelinesList(args []string, stdout, stderr io.Writer, opts Options) int {
|
|
||||||
fs := flag.NewFlagSet("pipelines list", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
configPath := fs.String("config", "", "config file path")
|
|
||||||
jsonOutput := fs.Bool("json", false, "write JSON output")
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
fmt.Fprintf(stderr, "notarius: unexpected argument %q\n", fs.Arg(0))
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg, _, err := loadConfig(*configPath, opts)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
if err := cfg.Validate(); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
ids := sortedPipelineIDs(cfg)
|
|
||||||
if *jsonOutput {
|
|
||||||
payload := struct {
|
|
||||||
Pipelines []string `json:"pipelines"`
|
|
||||||
}{Pipelines: ids}
|
|
||||||
encoded, err := json.Marshal(payload)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "notarius: marshal pipeline list: %v\n", err)
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
fmt.Fprintf(stdout, "%s\n", encoded)
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, id := range ids {
|
|
||||||
fmt.Fprintln(stdout, id)
|
|
||||||
}
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
func loadConfig(configPath string, opts Options) (config.Config, string, error) {
|
func loadConfig(configPath string, opts Options) (config.Config, string, error) {
|
||||||
path, err := discoverConfigPath(configPath, opts)
|
path, err := discoverConfigPath(configPath, opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
@@ -37,6 +38,9 @@ func TestRunControlsRejectSyntaxWithoutAllocatingState(t *testing.T) {
|
|||||||
{name: "blank debug directory", args: func(roots stateTestRoots) []string {
|
{name: "blank debug directory", args: func(roots stateTestRoots) []string {
|
||||||
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", ""}
|
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", ""}
|
||||||
}},
|
}},
|
||||||
|
{name: "missing configuration profile", args: func(roots stateTestRoots) []string {
|
||||||
|
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--config-profile"}
|
||||||
|
}},
|
||||||
{name: "debug directory without debug", args: func(roots stateTestRoots) []string {
|
{name: "debug directory without debug", args: func(roots stateTestRoots) []string {
|
||||||
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", filepath.Join(filepath.Dir(roots.debug), "requested-debug")}
|
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", filepath.Join(filepath.Dir(roots.debug), "requested-debug")}
|
||||||
}},
|
}},
|
||||||
@@ -79,7 +83,7 @@ func TestRecomputeStepCLIContract(t *testing.T) {
|
|||||||
{
|
{
|
||||||
name: "explicit step",
|
name: "explicit step",
|
||||||
configure: func(t *testing.T, roots stateTestRoots) {
|
configure: func(t *testing.T, roots stateTestRoots) {
|
||||||
replaceStateTestConfigLine(t, roots.config, " artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n", " steps:\n - id: chosen\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n")
|
replaceStateTestConfigLine(t, roots.config, " artifacts:\n items:\n extract:\n module: test/extract\n merge:\n module: test/merge\n normalize:\n module: test/normalize\n", " steps:\n - id: chosen\n artifacts:\n items:\n extract:\n module: test/extract\n merge:\n module: test/merge\n normalize:\n module: test/normalize\n")
|
||||||
},
|
},
|
||||||
flags: []string{"--resume", "--recompute-step", "chosen"},
|
flags: []string{"--resume", "--recompute-step", "chosen"},
|
||||||
wantCode: 0,
|
wantCode: 0,
|
||||||
@@ -161,7 +165,7 @@ func TestRunOnlyExecutesSelectedLanes(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
data = []byte(replaceRequiredOnce(t, string(data), " output: test/output\n", " other:\n extract: test/extract\n output: test/output\n"))
|
data = []byte(replaceRequiredOnce(t, string(data), " output:\n module: test/output\n", " other:\n extract:\n module: test/extract\n output:\n module: test/output\n"))
|
||||||
if err := os.WriteFile(roots.config, data, 0o600); err != nil {
|
if err := os.WriteFile(roots.config, data, 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -180,7 +184,7 @@ func TestRunOnlyExecutesSelectedLanes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
|
func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
|
||||||
t.Run("environment roots", func(t *testing.T) {
|
t.Run("retired environment roots have no effect", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
environmentOutput := filepath.Join(t.TempDir(), "environment-output")
|
environmentOutput := filepath.Join(t.TempDir(), "environment-output")
|
||||||
environmentDebug := filepath.Join(t.TempDir(), "environment-debug")
|
environmentDebug := filepath.Join(t.TempDir(), "environment-debug")
|
||||||
@@ -193,10 +197,10 @@ func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
|
|||||||
if result.code != 0 {
|
if result.code != 0 {
|
||||||
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
|
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
|
||||||
}
|
}
|
||||||
assertFile(t, filepath.Join(environmentOutput, filepath.Base(onlyChildDir(t, environmentOutput)), "result.json"))
|
assertFile(t, filepath.Join(roots.output, filepath.Base(onlyChildDir(t, roots.output)), "result.json"))
|
||||||
onlyChildDir(t, environmentDebug)
|
onlyChildDir(t, roots.debug)
|
||||||
assertAbsent(t, roots.output)
|
assertAbsent(t, environmentOutput)
|
||||||
assertAbsent(t, roots.debug)
|
assertAbsent(t, environmentDebug)
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("command flags override environment", func(t *testing.T) {
|
t.Run("command flags override environment", func(t *testing.T) {
|
||||||
@@ -227,23 +231,57 @@ func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
text := string(data)
|
text := string(data)
|
||||||
text = replaceRequiredOnce(t, text, fmt.Sprintf(" directory: %q\n", roots.output), "")
|
text = replaceRequiredOnce(t, text, fmt.Sprintf("output:\n directory: %q\n", roots.output), "")
|
||||||
text = replaceRequiredOnce(t, text, fmt.Sprintf(" directory: %q\n", roots.debug), "")
|
text = replaceRequiredOnce(t, text, fmt.Sprintf("debug:\n directory: %q\n", roots.debug), "")
|
||||||
if err := os.WriteFile(roots.config, []byte(text), 0o600); err != nil {
|
if err := os.WriteFile(roots.config, []byte(text), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
workDir := t.TempDir()
|
|
||||||
t.Chdir(workDir)
|
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
result := runWithStateRoots(t, roots, opts, nil)
|
result := runWithStateRoots(t, roots, opts, nil)
|
||||||
if result.code != 0 {
|
if result.code != 0 {
|
||||||
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
|
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
|
||||||
}
|
}
|
||||||
assertFile(t, filepath.Join(workDir, "notarius-output", filepath.Base(onlyChildDir(t, filepath.Join(workDir, "notarius-output"))), "result.json"))
|
configDirectory := filepath.Dir(roots.config)
|
||||||
onlyChildDir(t, filepath.Join(workDir, "notarius-debug"))
|
assertFile(t, filepath.Join(configDirectory, "notarius-output", filepath.Base(onlyChildDir(t, filepath.Join(configDirectory, "notarius-output"))), "result.json"))
|
||||||
|
onlyChildDir(t, filepath.Join(configDirectory, "notarius-debug"))
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunAppliesConfigurationProfileAndRecordsSelection(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
profileDirectory := filepath.Join(filepath.Dir(roots.config), "profiles")
|
||||||
|
profileOutput := filepath.Join(profileDirectory, "profile-output")
|
||||||
|
profileDebug := filepath.Join(profileDirectory, "profile-debug")
|
||||||
|
writeConfigKitFile(t, filepath.Join(profileDirectory, "development.yml"), "output:\n directory: ./profile-output\ndebug:\n directory: ./profile-debug\n")
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"run", "sample", "--config", roots.config, "--config-profile", "development",
|
||||||
|
"--input", roots.input, "--chunk_cache", "bypass", "--debug",
|
||||||
|
}, &stdout, &stderr, newStateTestHarness().options())
|
||||||
|
if code != 0 || stderr.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
assertFile(t, filepath.Join(profileOutput, filepath.Base(onlyChildDir(t, profileOutput)), "result.json"))
|
||||||
|
bundle := onlyChildDir(t, profileDebug)
|
||||||
|
var invocation debugbundle.Invocation
|
||||||
|
readStateTestSummaryJSON(t, bundle, "invocation.json", &invocation)
|
||||||
|
if invocation.ConfigProfile != "development" {
|
||||||
|
t.Fatalf("config profile = %q, want development", invocation.ConfigProfile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigProfileOmitsBaseSelection(t *testing.T) {
|
||||||
|
for _, profile := range []string{"", "default"} {
|
||||||
|
if got := effectiveConfigProfile(profile); got != "" {
|
||||||
|
t.Fatalf("effectiveConfigProfile(%q) = %q, want empty", profile, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := effectiveConfigProfile("development"); got != "development" {
|
||||||
|
t.Fatalf("effectiveConfigProfile(development) = %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||||
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
@@ -567,7 +605,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
data = []byte(replaceRequiredOnce(t, string(data), "extract: test/extract", "extract: test/failing-extract"))
|
data = []byte(replaceRequiredOnce(t, string(data), "module: test/extract", "module: test/failing-extract"))
|
||||||
if err := os.WriteFile(roots.config, data, 0o600); err != nil {
|
if err := os.WriteFile(roots.config, data, 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -92,7 +92,7 @@ func TestRunResultReportsSuccessfulRejection(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n validation_policy:\n semantic_rejection: reject_output\n"))
|
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize:\n module: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - module: generic/always_reject\n validation_policy:\n semantic_rejection: reject_output\n"))
|
||||||
if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil {
|
if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -258,7 +258,7 @@ func TestRunRedactsSensitiveModuleOptionsFromConfigAndPipelineSummaries(t *testi
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
configText := replaceRequiredOnce(t, string(data), " input: test/input\n", ` input:
|
configText := replaceRequiredOnce(t, string(data), " input:\n module: test/input\n", ` input:
|
||||||
module: test/input
|
module: test/input
|
||||||
options:
|
options:
|
||||||
api_key: CONFIG_SUMMARY_SECRET_SENTINEL
|
api_key: CONFIG_SUMMARY_SECRET_SENTINEL
|
||||||
@@ -276,7 +276,7 @@ func TestRunRedactsSensitiveModuleOptionsFromConfigAndPipelineSummaries(t *testi
|
|||||||
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
|
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
|
||||||
}
|
}
|
||||||
summaryRoot := filepath.Join(onlyChildDir(t, roots.debug), "summary")
|
summaryRoot := filepath.Join(onlyChildDir(t, roots.debug), "summary")
|
||||||
for _, name := range []string{"effective-config.json", "resolved-pipeline.json"} {
|
for _, name := range []string{"effective-config.yml", "resolved-pipeline.json"} {
|
||||||
contents, err := os.ReadFile(filepath.Join(summaryRoot, name))
|
contents, err := os.ReadFile(filepath.Join(summaryRoot, name))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -687,7 +687,7 @@ func newStateTestRoots(t *testing.T) stateTestRoots {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
roots.config = filepath.Join(base, "config.yml")
|
roots.config = filepath.Join(base, "config.yml")
|
||||||
config := fmt.Sprintf("version: 4\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
config := fmt.Sprintf("version: 5\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input:\n module: test/input\n chunk:\n module: test/chunk\n artifacts:\n items:\n extract:\n module: test/extract\n merge:\n module: test/merge\n normalize:\n module: test/normalize\n output:\n module: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
||||||
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -40,6 +40,9 @@ func normalizeDocumentPaths(document *Document, resolution *configkit.Resolution
|
|||||||
if profile.Output != nil {
|
if profile.Output != nil {
|
||||||
normalizeDocumentBindingReferencePaths(profile.Output, path.Key("output"), resolution, state)
|
normalizeDocumentBindingReferencePaths(profile.Output, path.Key("output"), resolution, state)
|
||||||
}
|
}
|
||||||
|
if documentStepsContainRelativeReferencePaths(profile.Steps) {
|
||||||
|
resolution.Record(path.Key("steps"), configkit.ChangeNormalization, "resolved configured paths in ordered steps")
|
||||||
|
}
|
||||||
for laneID, lane := range profile.Artifacts {
|
for laneID, lane := range profile.Artifacts {
|
||||||
normalizeDocumentLaneReferencePaths(&lane, path.Key("artifacts").Key(laneID), resolution, state)
|
normalizeDocumentLaneReferencePaths(&lane, path.Key("artifacts").Key(laneID), resolution, state)
|
||||||
profile.Artifacts[laneID] = lane
|
profile.Artifacts[laneID] = lane
|
||||||
@@ -57,6 +60,31 @@ func normalizeDocumentPaths(document *Document, resolution *configkit.Resolution
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func documentStepsContainRelativeReferencePaths(steps []PipelineStepDocument) bool {
|
||||||
|
for _, step := range steps {
|
||||||
|
if documentReferencesContainRelativePath(step.References) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, lane := range step.Artifacts {
|
||||||
|
if documentReferencesContainRelativePath(lane.Extract.References) ||
|
||||||
|
(lane.Merge != nil && documentReferencesContainRelativePath(lane.Merge.References)) ||
|
||||||
|
(lane.Normalize != nil && documentReferencesContainRelativePath(lane.Normalize.References)) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func documentReferencesContainRelativePath(references map[string]ReferenceDocument) bool {
|
||||||
|
for _, source := range references {
|
||||||
|
if source.Path != nil && *source.Path != "" && !filepath.IsAbs(*source.Path) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
func initializeDocumentMainDirectory(resolution *configkit.Resolution, state *documentLoadState) {
|
func initializeDocumentMainDirectory(resolution *configkit.Resolution, state *documentLoadState) {
|
||||||
if state == nil || state.mainDirectory != "" {
|
if state == nil || state.mainDirectory != "" {
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ type Invocation struct {
|
|||||||
InputPath string `json:"input_path,omitempty"`
|
InputPath string `json:"input_path,omitempty"`
|
||||||
ConfigPath string `json:"config_path,omitempty"`
|
ConfigPath string `json:"config_path,omitempty"`
|
||||||
ConfigSource string `json:"config_source,omitempty"`
|
ConfigSource string `json:"config_source,omitempty"`
|
||||||
|
ConfigProfile string `json:"config_profile,omitempty"`
|
||||||
OnlyLanes []string `json:"only_lanes,omitempty"`
|
OnlyLanes []string `json:"only_lanes,omitempty"`
|
||||||
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
||||||
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
|
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
|
||||||
|
|||||||
Reference in New Issue
Block a user