Compare commits
16 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| aae01a59ce | |||
| 71cf084e8b | |||
| e327a5eef7 | |||
| af62b3bacc | |||
| e1097cc7a5 | |||
| 3f226c91d0 | |||
| 64509947df | |||
| 29c7c165a5 | |||
| 4ef69314e1 | |||
| 1d1434b118 | |||
| f4779fede9 | |||
| 01c08ec8e4 | |||
| d7ea0fd32d | |||
| ec99d48619 | |||
| 9e74df0b6e | |||
| 6e5e277426 |
@@ -40,6 +40,21 @@ have no effect; `NOTARIUS_CONFIG` remains a source-discovery input.
|
||||
[LLM profile](../examples/llm-profiles/dnd-extraction.yml) and imports its
|
||||
pipeline from [a fragment](../examples/pipelines/dnd-session.yml).
|
||||
|
||||
The complete fragment also defines `dnd-session-development`, which inherits
|
||||
the complete pipeline and profile while setting
|
||||
`structured_output_repair_attempts: 0`. Select it with:
|
||||
|
||||
```sh
|
||||
notarius run dnd-session-development \
|
||||
--config /path/to/dnd-complete.config.yml \
|
||||
--input /path/to/transcript.json
|
||||
```
|
||||
|
||||
See the [CLI reference](cli.md#run) for the full invocation contract. This
|
||||
variant disables PromptKit structural repairs without changing configured producer
|
||||
retry budgets. When using another profile, operators can instead select
|
||||
`dnd-session` and set its `llm_profile` to that profile.
|
||||
|
||||
Use these complete files as starting points rather than combining the
|
||||
illustrative fragments in this reference.
|
||||
|
||||
@@ -80,10 +95,12 @@ umask, setgid inheritance, and default ACLs; see [Operations](operations.md#cach
|
||||
|
||||
Every nonempty relative configured destination or reference path resolves from
|
||||
the canonical directory of the file that supplies its surviving declaration.
|
||||
This includes imported fragments and selected configuration overlays. Built-in
|
||||
relative destinations use the selected main configuration file's directory;
|
||||
explicit `--output-dir` and `--debug-dir` values use the invocation working
|
||||
directory. Absolute values remain unchanged.
|
||||
This includes imported fragments, selected configuration overlays, and values
|
||||
copied through inheritance: the copied value keeps its original declaring
|
||||
file. A child-supplied replacement resolves from the child's declaring file.
|
||||
Built-in relative destinations use the selected main configuration file's
|
||||
directory; explicit `--output-dir` and `--debug-dir` values use the invocation
|
||||
working directory. Absolute values remain unchanged.
|
||||
|
||||
The root file can use ConfigKit imports to compose disjoint definitions. Imports
|
||||
and selected overlays remain under the configuration root; they do not merge or
|
||||
@@ -91,6 +108,50 @@ append a pipeline's `steps` sequence. An overlay replaces sequences as defined
|
||||
by ConfigKit, so changing an individual ordered binding requires supplying the
|
||||
whole replacement sequence.
|
||||
|
||||
## Sibling Inheritance
|
||||
|
||||
ConfigKit v0.5.0 supports native `extends` composition in mappings. A directive
|
||||
names exactly one immediate sibling in the same mapping; it is not a path or a
|
||||
cross-scope lookup. Imports, the selected overlay, and explicit overrides are
|
||||
composed before inheritance, which runs before typed decoding and normal
|
||||
Notarius validation. Every definition, including a parent and definitions not
|
||||
selected for a run, remains subject to ordinary validation.
|
||||
|
||||
Child mappings recursively overlay parent mappings. A child sequence, scalar,
|
||||
or null replaces the inherited value. This makes pipeline inheritance useful
|
||||
for reusing a complete pipeline while replacing whole ordered lists:
|
||||
|
||||
~~~yaml
|
||||
pipelines:
|
||||
production:
|
||||
input: {module: seriatim}
|
||||
llm_profile: dnd-extraction
|
||||
# Other complete pipeline settings and ordered steps...
|
||||
development:
|
||||
extends: production
|
||||
structured_output_repair_attempts: 0
|
||||
~~~
|
||||
|
||||
Pipeline names are siblings within `pipelines`; nested mappings such as named
|
||||
artifact lanes also have their own sibling namespace. Step entries are a
|
||||
sequence, so a step cannot extend another step. Replacing `steps` or a validator
|
||||
chain supplies the complete replacement sequence; entries are not appended or
|
||||
merged by identifier.
|
||||
|
||||
Inherited values count as explicit input for Notarius's existing
|
||||
presence-sensitive rules: zero, false, and empty-list overrides stay explicit
|
||||
instead of restoring a default. Null also replaces the inherited value, but
|
||||
each field's ordinary type and validation rules still apply; null is not
|
||||
universally valid. Omitting `extends` preserves a relationship supplied by a
|
||||
lower-priority source, while `extends: null` cancels that relationship. It
|
||||
does not independently clear other inherited fields.
|
||||
|
||||
`extends` is now a reserved key in every input mapping, including dynamic
|
||||
module options and named maps. Existing ordinary data or entry names using that
|
||||
key must be renamed along with any consumer that reads them. Configuration
|
||||
overlays, pipeline inheritance, and PromptKit profile `base_profile`
|
||||
inheritance are separate mechanisms.
|
||||
|
||||
## PromptKit Profiles
|
||||
|
||||
The optional **promptkit** object selects one source of profile definitions and
|
||||
|
||||
@@ -2,17 +2,18 @@
|
||||
|
||||
Notarius uses the public root package
|
||||
[`gitea.maximumdirect.net/eric/ConfigKit`](https://gitea.maximumdirect.net/eric/ConfigKit)
|
||||
at v0.4.2 for version 5 configuration acquisition, imports, overlays, explicit
|
||||
overrides, diagnostics, and safe inspection rendering.
|
||||
at v0.5.0 for version 5 configuration acquisition, imports, overlays, explicit
|
||||
overrides, native sibling inheritance, diagnostics, and safe inspection
|
||||
rendering. ConfigKit composes source precedence and inheritance before typed
|
||||
decoding; Notarius does not perform a second inheritance merge.
|
||||
|
||||
Notarius owns the typed document, domain defaults and validation, runtime
|
||||
conversion, catalog-dependent pipeline resolution, source-aware configured
|
||||
paths, and command discovery. It uses only ConfigKit's public path, result,
|
||||
diagnostic, and resolution-source APIs. Upstream documentation owns general
|
||||
ConfigKit semantics; [Configuration](../config.md) owns Notarius's file and
|
||||
operator contract.
|
||||
operator contract. See the pinned upstream [v0.5.0 release notes](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/releases/v0.5.0.md)
|
||||
and [configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/config.md).
|
||||
|
||||
The v0.4.2 patch updates upstream release tooling and policy/licensing
|
||||
documentation without changing configuration behavior or the public API. See
|
||||
the canonical [ConfigKit attribution](../oss/attribution.md#configkit) for its
|
||||
license and notices.
|
||||
See the canonical [ConfigKit attribution](../oss/attribution.md#configkit) for
|
||||
its license and notices.
|
||||
|
||||
@@ -15,7 +15,7 @@ runs anything.
|
||||
|
||||
| Boundary | Inputs | Outputs | Does not own |
|
||||
| --- | --- | --- | --- |
|
||||
| Loading | Selected file path, profile, overrides, and invocation directory | **LoadedConfig** with ConfigKit inspection metadata and a populated **Config** | Choosing the file path or reporting a command result. |
|
||||
| Loading | Selected file path, overlay, overrides, and invocation directory | **LoadedConfig** with ConfigKit inspection metadata and a populated **Config** | Choosing the file path or reporting a command result. |
|
||||
| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
|
||||
| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
|
||||
| Inspection | **LoadedConfig** and an optional JSON Pointer | Redacted effective YAML or explanation text | Pipeline selection, module preparation, or provider work. |
|
||||
@@ -28,10 +28,21 @@ at their respective [CLI](cli.md), [pipeline](pipeline.md), and
|
||||
|
||||
## Loading And Validation
|
||||
|
||||
ConfigKit acquires the selected root, imports, optional selected overlay, and
|
||||
explicit CLI overrides in precedence order. Its document definition supplies
|
||||
typed defaults, source-aware path normalization, and structural validation
|
||||
before conversion to the independently owned runtime **Config**. Only
|
||||
ConfigKit composes the selected root, imports, optional selected overlay, and
|
||||
explicit CLI overrides in precedence order, then expands native sibling
|
||||
inheritance before typed decoding. The document definition supplies typed
|
||||
defaults and structural validation; Notarius's recorded resolution changes
|
||||
normalize configured paths after decode. Path normalization follows the source
|
||||
of the surviving value, including a value copied through inheritance, rather
|
||||
than the inheritance directive's location. Inherited input remains explicit
|
||||
for presence-sensitive behavior, including zero, false, and empty values;
|
||||
existing field validation still governs whether the resulting values are valid.
|
||||
ConfigKit's inspection history retains inheritance transfers alongside later
|
||||
normalization, and effective configuration renders the normalized values used
|
||||
by runtime conversion. The private typed document consumes the directive, so
|
||||
no upstream inheritance type or unresolved directive reaches runtime pipeline
|
||||
or module contracts. See [Configuration](../config.md#sibling-inheritance)
|
||||
for the operator syntax and field rules. Only
|
||||
`NOTARIUS_CONFIG` participates in discovery; retired field-level environment
|
||||
variables do not participate in loading. Version 5 decoding rejects scalar
|
||||
module bindings and scalar external references. ConfigKit retains redacted,
|
||||
|
||||
@@ -18,7 +18,7 @@ License and notices: [CC BY-SA 4.0](licenses/CC-BY-SA-4.0.txt) |
|
||||
|
||||
## ConfigKit
|
||||
|
||||
Selected module: `gitea.maximumdirect.net/eric/ConfigKit` at `v0.4.2`.
|
||||
Selected module: `gitea.maximumdirect.net/eric/ConfigKit` at `v0.5.0`.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
@@ -26,10 +26,10 @@ SPDX-License-Identifier: LGPL-3.0-or-later
|
||||
|
||||
License and notices: [LGPL 3.0](licenses/LGPL-3.0.txt) |
|
||||
[GPL 3.0](licenses/GPL-3.0.txt) |
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/67c03690181c5af8dde6be325abac241dae6732f/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/67c03690181c5af8dde6be325abac241dae6732f/LICENSE).
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/LICENSE).
|
||||
|
||||
[Source](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/67c03690181c5af8dde6be325abac241dae6732f).
|
||||
[Source](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd).
|
||||
|
||||
## PromptKit
|
||||
|
||||
|
||||
229
docs/roadmap/configkit.md
Normal file
229
docs/roadmap/configkit.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# ConfigKit 0.5 And Configuration Inheritance
|
||||
|
||||
## Purpose
|
||||
|
||||
Adopt ConfigKit v0.5.0 and make its sibling inheritance useful and supported in
|
||||
Notarius configuration, particularly for deriving development or testing
|
||||
pipelines from a complete D&D pipeline without duplicating the pipeline body.
|
||||
|
||||
ConfigKit owns source composition and inheritance. Notarius continues to own
|
||||
typed configuration, domain defaults and validation, pipeline resolution, and
|
||||
execution. This work does not introduce a second configuration-merging engine.
|
||||
|
||||
The [implementation plan](implementation.md) owns the ordered work and
|
||||
validation sequence. Release preparation and publication remain separate.
|
||||
|
||||
## Reviewed Baseline
|
||||
|
||||
Reviewed on 2026-10-03. Notarius pins ConfigKit v0.4.2 and uses its ordinary
|
||||
`Load` API in `internal/core/config/load.go`. Configuration version 5 supports
|
||||
imports, selected configuration overlays, explicit command overrides, recorded
|
||||
normalization, and redacted effective/explain output. Runtime configuration is
|
||||
detached from the ConfigKit inspection snapshot.
|
||||
|
||||
Named pipelines and artifact lanes are mappings; pipeline `steps` and validator
|
||||
chains are sequences. The complete example imports
|
||||
`examples/pipelines/dnd-session.yml`, while the existing development overlay
|
||||
changes operational settings. Relative configured paths are resolved from the
|
||||
file supplying their surviving declaration. The current path resolver uses
|
||||
ConfigKit origin/source metadata and host filesystem paths.
|
||||
|
||||
Target ConfigKit v0.5.0 at full source commit
|
||||
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`. Consult the pinned
|
||||
[release guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
|
||||
and [configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/config.md).
|
||||
The release adds inheritance, layered loading, filesystem-backed sources,
|
||||
mapping sources, and passive resource contexts. Its Go requirement and YAML
|
||||
dependency are unchanged. The upgrade has not yet been trialed against
|
||||
Notarius's test suite at this review baseline.
|
||||
|
||||
## Target End State
|
||||
|
||||
### Native Sibling Inheritance
|
||||
|
||||
Operators can define a named pipeline with `extends: parent-pipeline` and
|
||||
override only the settings that differ. The parent remains an ordinary named,
|
||||
valid pipeline; no abstract-template flag, template registry, or special
|
||||
validation exemption is introduced. Derived pipelines use the existing run,
|
||||
listing, validation, effective-configuration, and explanation commands.
|
||||
|
||||
Use ConfigKit's native semantics without restricting inheritance to pipelines
|
||||
or implementing a Notarius-specific subset:
|
||||
|
||||
- `extends` selects one exact key among the mapping's immediate siblings.
|
||||
It is not a path, cross-scope lookup, or multiple-parent expression.
|
||||
- Source precedence is assembled before inheritance, and inheritance runs
|
||||
before typed decoding and application resolution. Inherited input is explicit
|
||||
for presence-sensitive defaults, including zero, false, empty, and null.
|
||||
- Child mappings recursively overlay parent mappings; sequences, scalars,
|
||||
and null replace inherited values. Existing typed/domain validation still
|
||||
determines whether the resulting value is valid.
|
||||
- Omitting `extends` retains a lower-priority relationship; `extends: null`
|
||||
cancels the relationship, not arbitrary inherited fields independently.
|
||||
- Root and sequence-element mappings have no sibling namespace for their own
|
||||
string relationship. Nested mappings may have one. In particular, individual
|
||||
entries of the `steps` sequence cannot extend other step entries.
|
||||
- ConfigKit rejects missing/non-mapping parents, invalid directive types, and
|
||||
cycles, and resolves effective relationships even in unselected entries.
|
||||
- Parent definitions receive normal typed decoding and Notarius validation.
|
||||
Inheritance is not a way to hide invalid partial pipeline definitions.
|
||||
- Inherited nested relationships resolve at their destination according to
|
||||
ConfigKit's rules; do not pre-expand or reinterpret them in Notarius.
|
||||
|
||||
Names are matched by ConfigKit before Notarius's existing identifier trimming.
|
||||
Document exact-key lookup rather than introducing pre-inheritance name rewriting.
|
||||
|
||||
The principal documented workflow is whole-pipeline reuse. Inheritance does not
|
||||
append steps, merge sequences by `id`, or allow a child to override one list
|
||||
element without supplying a replacement list. The same applies to validator
|
||||
chains. Do not disguise this boundary through custom list-merging code.
|
||||
|
||||
### Paths, Provenance, And Runtime Boundaries
|
||||
|
||||
An inherited relative reference or resource path remains relative to its
|
||||
original declaring file, not the child's file or the invocation directory.
|
||||
Imported parents, overlays, and nested inheritance must retain that behavior.
|
||||
Direct child declarations use their own source context. Built-in relative
|
||||
destinations and CLI-supplied paths keep their existing bases.
|
||||
|
||||
Review `documentPathBase` and related original-path/normalization handling
|
||||
against the new origin metadata. Make narrowly required corrections if needed;
|
||||
do not assume that successful typed decoding proves correct path resolution.
|
||||
Effective output must agree with the runtime paths, and explanations must retain
|
||||
the upstream derivation evidence through recorded Notarius normalization.
|
||||
|
||||
Keep the ConfigKit boundary private: no upstream inheritance objects or
|
||||
unresolved directives reach runtime pipeline/module types. ConfigKit expands
|
||||
relationships once during loading. Notarius's existing binding-profile,
|
||||
validation-policy, and module-default resolution still operate on the resulting
|
||||
configuration; they are not replaced with generic sibling inheritance.
|
||||
|
||||
Configuration overlays, pipeline inheritance, and PromptKit `base_profile`
|
||||
inheritance remain distinct mechanisms. The latter still belongs to PromptKit.
|
||||
|
||||
### Compatibility And Upgrade Policy
|
||||
|
||||
Keep Notarius configuration `version: 5`. The typed runtime shape is unchanged;
|
||||
the new capability belongs to configuration composition. No new CLI flag or
|
||||
inheritance opt-in setting is needed or available upstream.
|
||||
|
||||
The significant compatibility change is that `extends` becomes a reserved key
|
||||
in every input mapping, including dynamic module-option maps and named maps.
|
||||
It is consumed before typed decoding, so it can no longer be ordinary data or
|
||||
an ordinary entry name there. Audit maintained configuration/fixtures for such
|
||||
usage and document that affected operator configurations must rename the key
|
||||
or identifier and any associated consumer field. Do not silently strip, alias,
|
||||
or bypass the directive to preserve an incompatible interpretation.
|
||||
|
||||
Configurations without that collision should retain their existing meaning.
|
||||
Preserve discovery, import rules, overlay selection, explicit CLI override
|
||||
precedence, strict decoding, redaction, defaults, and validation. Existing
|
||||
configuration-stage inheritance does not alter the fixed execution pipeline,
|
||||
artifact contracts, warning policy, or checkpoint identity design.
|
||||
|
||||
The eventual Notarius release should be a new minor version, explaining both
|
||||
the feature and the newly reserved input key. Release preparation/publication
|
||||
is separate from this work; no exact release number is selected here.
|
||||
|
||||
## Required Work
|
||||
|
||||
### Dependency And Configuration Integration
|
||||
|
||||
Upgrade `go.mod` and `go.sum` to the exact target. Keep PromptKit and unrelated
|
||||
dependencies unchanged unless dependency resolution demonstrably requires it.
|
||||
Continue using ordinary `configkit.Load`, the existing typed defaults factory,
|
||||
and source-aware resolution/validation callbacks.
|
||||
|
||||
Inspect the load, operational-document, path, conversion, inspection, and CLI
|
||||
boundaries. Add only integration adjustments demonstrated necessary by the
|
||||
upstream change. There should be no `Extends` field added to Notarius runtime
|
||||
types and no homegrown inheritance algorithm.
|
||||
|
||||
Update ConfigKit's factual attribution version and COPYRIGHT/LICENSE/source
|
||||
links to the target commit after inspecting its actual notices. Preserve the
|
||||
selected project grant and reuse unchanged local license texts.
|
||||
|
||||
### Maintained Example
|
||||
|
||||
Extend the complete example's imported pipeline definitions with a small
|
||||
`dnd-session-development` sibling deriving from `dnd-session`, demonstrating a
|
||||
pipeline-level setting override such as a smaller structural-repair budget.
|
||||
Use already supported values and existing profile sources so the example
|
||||
validates without an invented model/profile dependency. Explain that
|
||||
`llm_profile` is another useful override when the operator supplies that profile.
|
||||
|
||||
Retain the runnable parent, minimal example, and existing operational overlay.
|
||||
Do not duplicate the complete steps list or add a third standalone configuration
|
||||
merely to demonstrate inheritance. Update example-loading/listing expectations
|
||||
for the additional pipeline, checking inherited behavior rather than snapshots
|
||||
of incidental formatting or total line counts.
|
||||
|
||||
### Canonical Documentation
|
||||
|
||||
- `docs/config.md`: own syntax, composition order, exact sibling lookup,
|
||||
replacement/null semantics, path bases, reserved-key migration, and limits
|
||||
on step/validator-list composition. Link to the maintained example and pinned
|
||||
upstream rules rather than duplicating the entire upstream manual.
|
||||
- `docs/internal/configuration.md`: describe expansion before typed decoding,
|
||||
presence and source-aware normalization, and the unchanged runtime boundary.
|
||||
- `docs/integrations/pkg-configkit.md`: update the supported version, public
|
||||
contract links, and integration scope.
|
||||
- `docs/oss/attribution.md`: update the dependency's verified source/notice links.
|
||||
- `docs/cli.md` and consumer guides: update only where navigation or examples
|
||||
benefit from showing selection of the derived pipeline. Do not invent new
|
||||
invocation syntax or duplicate the configuration reference.
|
||||
|
||||
The existing ConfigKit architecture decision remains applicable. No new ADR is
|
||||
required solely for consuming the library's composition feature; do not rewrite
|
||||
an accepted ADR or turn new private implementation details into invariants.
|
||||
|
||||
## Verification And Acceptance
|
||||
|
||||
Use focused Notarius boundary tests, not a duplicate of ConfigKit's full suite:
|
||||
|
||||
- Existing maintained configurations continue loading and resolving.
|
||||
- A derived pipeline retains its own identifier and inherits the complete
|
||||
parent pipeline while applying its explicit overrides.
|
||||
- Imports and selected overlays participate in composition before inheritance;
|
||||
explicit CLI overrides keep their documented precedence.
|
||||
- Representative zero/false/null inputs exercise existing presence-sensitive
|
||||
defaulting or validation correctly; null is not universally valid merely
|
||||
because inheritance can represent it.
|
||||
- Replacing a steps or validator sequence does not concatenate or merge it.
|
||||
- References inherited from a parent in another file retain their declaring
|
||||
directory, while child-supplied replacements use the child's directory.
|
||||
- Effective and explanation output remain redacted and expose useful source
|
||||
and inheritance information after application normalization.
|
||||
- Representative invalid inheritance produces ConfigKit load diagnostics through
|
||||
the normal Notarius CLI error path, without a parallel validation engine.
|
||||
- Maintained examples and their imported fragments validate offline without
|
||||
credentials or provider calls.
|
||||
|
||||
Configuration and CLI boundary coverage should protect document/path/operational,
|
||||
effective/explanation, and maintained-example behavior. Framework-resolution
|
||||
coverage is appropriate where a distinct handoff risk requires it. Acceptance
|
||||
includes the offline default suite, vet, and build with `GOWORK=off`, dependency
|
||||
tidiness, and documentation diff/link checks appropriate to the changes. The
|
||||
implementation plan owns their sequencing; these checks are not additional
|
||||
permanent release gates.
|
||||
|
||||
## Deferred Capabilities And Non-Goals
|
||||
|
||||
- Do not adopt `LoadLayered` or add a new underlay precedence tier now. Existing
|
||||
imports, overlays, and typed defaults remain the loading model. Bundled
|
||||
pipeline presets or site-wide default layers need a separate operator-facing
|
||||
design if selected later.
|
||||
- Do not move Go defaults into embedded YAML or mapping sources solely because
|
||||
v0.5.0 provides those APIs.
|
||||
- Do not generalize references to arbitrary `fs.FS` resources in this work.
|
||||
`SourceContext` may be used if it materially helps the required path behavior,
|
||||
but its filesystem access rules must not silently replace Notarius's existing
|
||||
host-path contract or introduce new path restrictions.
|
||||
- No abstract templates, multiple inheritance, list-element composition,
|
||||
new template schema, new pipeline scheduler, or extra configuration knobs.
|
||||
- No opt-in gate or Notarius-only prohibition on other valid native inheritance
|
||||
locations; use upstream semantics and existing domain validation.
|
||||
|
||||
There are no significant open design decisions at this baseline. Implementation
|
||||
should verify the new metadata behavior and correct concrete integration gaps
|
||||
without broadening the agreed feature scope.
|
||||
528
docs/roadmap/implementation.md
Normal file
528
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,528 @@
|
||||
# ConfigKit Inheritance Implementation Plan
|
||||
|
||||
## Purpose And Scope
|
||||
|
||||
Implement [ConfigKit 0.5 And Configuration Inheritance](configkit.md). That
|
||||
feature roadmap owns the intended behavior and policy choices. This document
|
||||
owns the ordered implementation work. Each numbered stage is one prompt for a
|
||||
Luna coding agent; complete its acceptance criteria before starting the next.
|
||||
|
||||
The starting baseline pins ConfigKit v0.4.2 and PromptKit v0.11.2, uses ordinary
|
||||
`configkit.Load`, and accepts Notarius configuration version 5. Upgrade only
|
||||
ConfigKit to v0.5.0, source commit
|
||||
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`. The release tag is annotated: use
|
||||
`git rev-parse 'v0.5.0^{commit}'` when verifying the source commit in a checkout.
|
||||
|
||||
The implementation adds supported native configuration inheritance and a
|
||||
maintained example. It does not require changing pipeline execution, artifact
|
||||
schemas, or release tooling. Release preparation and publication remain a
|
||||
separate task; this feature belongs in a future minor release.
|
||||
|
||||
## Working Instructions For Every Stage
|
||||
|
||||
- Read `docs/development.md`, `docs/policy/agents.md`, and the feature roadmap.
|
||||
Use the development guide to select additional applicable policies, including
|
||||
architecture, testing, documentation, and OSS for the changes being made.
|
||||
- Inspect the named code and tests before editing. Prefer the repository's
|
||||
code knowledge graph for code discovery. Inspect the pinned upstream release
|
||||
and public API when an integration detail needs confirmation; do not assume
|
||||
a sibling checkout's current branch equals the released version.
|
||||
- Keep ordinary `configkit.Load`, typed defaults, and the existing private
|
||||
configuration boundary. ConfigKit handles `extends` before decoding; add no
|
||||
`Extends` field, second merge pass, application template registry, or opt-in
|
||||
flag. Accept valid native inheritance locations under existing domain rules.
|
||||
- Make production changes only where a concrete integration failure requires
|
||||
them. A stage may consist of focused tests and documentation when the
|
||||
existing adapter already behaves correctly. Fix a regression rather than
|
||||
weakening an established expectation to accommodate the upgrade.
|
||||
- Test Notarius's boundary with the real ConfigKit dependency and small,
|
||||
deterministic fixtures. Reuse existing helpers and coverage; do not replicate
|
||||
ConfigKit's complete test matrix, add diagnostic snapshots, or assert
|
||||
incidental output formatting. Keep the default suite offline and independent
|
||||
of credentials and model calls.
|
||||
- Use `GOWORK=off` for Go commands so a sibling checkout cannot substitute for
|
||||
the pinned modules. Update canonical documentation as each behavior lands.
|
||||
Review the diff, run the stage's checks, and commit the coherent result under
|
||||
the agent policy. Record any unresolved failure in the handoff rather than
|
||||
marking that stage complete. Do not push or publish a release as part of this
|
||||
plan.
|
||||
|
||||
## Stage 1 — Upgrade The Dependency And Its Notices
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: establish the released dependency baseline without redesigning the
|
||||
configuration adapter.
|
||||
|
||||
Inspect:
|
||||
|
||||
- `go.mod`, `go.sum`, and `internal/core/config/load.go`.
|
||||
- `docs/integrations/pkg-configkit.md` and the ConfigKit entry in
|
||||
`docs/oss/attribution.md`.
|
||||
- ConfigKit's pinned `docs/releases/v0.5.0.md`, `go.mod`, `COPYRIGHT`, and
|
||||
`LICENSE`; consult the sibling checkout with `git show v0.5.0:<path>`.
|
||||
|
||||
Work:
|
||||
|
||||
1. Run `GOWORK=off go get gitea.maximumdirect.net/eric/ConfigKit@v0.5.0`, then
|
||||
`GOWORK=off go mod tidy`. Review the selected dependency graph. Retain
|
||||
PromptKit v0.11.2 and unrelated pins unless module resolution requires a
|
||||
change; explain any such change in the handoff.
|
||||
2. Search maintained configuration, fixtures, YAML tags, and documentation for
|
||||
ordinary uses of the key or entry name `extends`. For example, use
|
||||
`rg -n '\bextends\b' examples internal docs assets`. Classify any matches:
|
||||
new inheritance instructions are valid; ordinary field/key uses collide
|
||||
with the new reserved directive. Rename actual maintained collisions and
|
||||
their consumers together, without an alias or bypass. Do not rename the
|
||||
unrelated PromptKit `base_profile` setting.
|
||||
3. Run focused existing tests. Address compile/API compatibility failures with
|
||||
the smallest necessary adapter changes. Native inheritance needs no
|
||||
registration or load option.
|
||||
4. Update the integration document's pin and brief description to v0.5.0;
|
||||
replace the v0.4.2 patch-only explanation with the new composition boundary.
|
||||
Link to pinned upstream release/configuration documentation.
|
||||
5. Update only ConfigKit's attribution version and exact source/notice links
|
||||
to the target commit, preserving the verified copyright and
|
||||
`LGPL-3.0-or-later` expression. Reuse the existing LGPL/GPL texts when
|
||||
unchanged. Do not refresh unrelated component attribution or policies.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The declared and selected ConfigKit module is v0.5.0, without a workspace or
|
||||
local `replace` dependency.
|
||||
- Existing configuration and CLI tests pass; maintained ordinary `extends`
|
||||
collisions, if any, have been resolved explicitly.
|
||||
- Integration and attribution documents agree with the actual dependency and
|
||||
imported source commit. The adapter still uses ordinary `Load`.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go list -m gitea.maximumdirect.net/eric/ConfigKit
|
||||
GOWORK=off go test ./internal/core/config ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Prompt size: one dependency upgrade, its notices, and demonstrated compatibility
|
||||
adjustments. Leave new inheritance coverage to the following stages.
|
||||
|
||||
## Stage 2 — Prove Pipeline Composition And Runtime Conversion
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: establish whole-pipeline reuse through the real configuration boundary.
|
||||
|
||||
Inspect `load.go`, `document.go`, `loaded_config.go`, `pipeline_document.go`,
|
||||
`document_test.go`, `pipeline_document_test.go`, and
|
||||
`effective_config_contract_test.go` under `internal/core/config`. Reuse the
|
||||
existing test catalog for runtime resolution.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add focused loading coverage in `internal/core/config/inheritance_test.go`.
|
||||
Define a valid parent pipeline and a child with `extends: parent` plus a
|
||||
pipeline-level override. Assert the inherited input, bindings, and ordered
|
||||
steps survive conversion, while the child retains its own pipeline ID and
|
||||
the parent keeps its own settings.
|
||||
Keep the fixture small rather than copying the complete D&D example.
|
||||
2. Resolve the resulting child with the existing catalog. Assert its selected
|
||||
ID and overridden setting reach the resolved pipeline. Use the catalog's
|
||||
registered module keys and avoid generated-reference requirements in this
|
||||
small fixture; the full generated-reference handoff is covered in Stage 7.
|
||||
Reuse existing resolution tests for unrelated defaults and digests.
|
||||
3. Add one replacement case: the child supplies a complete different `steps`
|
||||
list. Assert replacement and ordering, with no parent steps appended or
|
||||
merged by ID. Both resulting pipelines must satisfy ordinary validation.
|
||||
4. Add one native nested-mapping case using sibling artifact lanes in an
|
||||
`artifacts` mapping: a child lane inherits a valid parent lane and overrides
|
||||
a binding value. This proves that Notarius does not limit inheritance to
|
||||
pipeline mappings. Other upstream-supported mapping shapes need no custom
|
||||
permission mechanism.
|
||||
5. In `docs/config.md`, add the canonical inheritance explanation: exact
|
||||
immediate-sibling lookup, one parent, source composition before expansion,
|
||||
recursive mapping combination, sequence/scalar/null replacement, normal
|
||||
validation of parent and child, and all effective relationships being
|
||||
resolved. State that step-sequence entries cannot extend each other.
|
||||
Include a small illustrative pipeline snippet and the reserved `extends`
|
||||
migration note. Distinguish configuration overlays and PromptKit profiles.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Loading and resolving a child yields an ordinary independently identified
|
||||
pipeline with the expected inherited composition and explicit override.
|
||||
- Replacement lists retain only the child list, in its configured order.
|
||||
- Valid nested artifact-lane inheritance works through native ConfigKit
|
||||
behavior without extra runtime types or application merging.
|
||||
- Configuration version remains 5 and the user reference describes the
|
||||
implemented syntax and reserved-key compatibility impact.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/core/config
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Prompt size: small composition fixtures, runtime handoff, and one canonical
|
||||
syntax section. Presence-sensitive cases belong to Stage 3.
|
||||
|
||||
## Stage 3 — Preserve Presence And Existing Domain Validation
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: ensure inherited input remains explicit through Notarius normalization,
|
||||
defaulting, and validator-chain conversion.
|
||||
|
||||
Inspect `operational_document.go`, `pipeline_document.go`,
|
||||
`document_normalizer.go`, `loaded_config.go`, and their existing presence tests.
|
||||
Extend `inheritance_test.go` rather than adding a parallel fixture framework.
|
||||
|
||||
Work:
|
||||
|
||||
1. Cover inherited `structured_output_repair_attempts: 0`: it remains a
|
||||
populated zero override after loading and resolution, rather than reverting
|
||||
to a pipeline or prompt default. Also cover a child explicitly overriding a
|
||||
nonzero parent budget with zero.
|
||||
2. Cover an inherited supported `validators: []` on a chunk or extraction
|
||||
binding. Its runtime override retains `Set=true` and an empty chain. Cover
|
||||
a child replacing a nonempty parent chain with `[]`; do not concatenate it
|
||||
or reinstall default validators because the child list is empty.
|
||||
3. Cover an inherited false module-option value, such as
|
||||
`output.options.evidence_context.enabled: false`, remaining false in runtime
|
||||
configuration. Inspect the decoded document's input metadata to establish
|
||||
explicit presence where needed, rather than relying on truthiness.
|
||||
4. Cover inherited or child-supplied null for a field already rejected when
|
||||
explicit, such as `structured_output_repair_attempts`. Require the existing
|
||||
contextual load failure. Inheritance's support for null does not make that
|
||||
field nullable. Reuse existing direct-input null coverage.
|
||||
5. Use a small overlay fixture to demonstrate the sequencing: an overlay that
|
||||
changes the parent before expansion changes the child's inherited value;
|
||||
a child's own explicit replacement wins over the parent's value. Verify
|
||||
`extends: null` cancellation using a child with its own valid input binding,
|
||||
and verify omission of the directive in an overlay preserves the base
|
||||
relationship. An explicit override record for the parent value must be
|
||||
reflected in an inheriting child as well.
|
||||
6. If any integration defect appears, correct Notarius's existing presence or
|
||||
conversion handling, using ConfigKit's metadata. Add no application
|
||||
inheritance evaluator. Refine `docs/config.md` to explain explicit
|
||||
zero/false/empty/null and cancellation without promising universal validity.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Inherited and direct replacements retain their existing explicit-value
|
||||
semantics, including zero repair budgets and empty validator overrides.
|
||||
- Existing null restrictions are neither relaxed nor expanded.
|
||||
- Overlay/explicit-override composition precedes inheritance, and cancellation
|
||||
behaves as described by the pinned upstream contract.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/core/config
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Prompt size: presence and precedence at one package boundary. This stage does
|
||||
not change defaults or add CLI flags for arbitrary configuration overrides.
|
||||
|
||||
## Stage 4 — Verify Declaration-Relative Paths Through Inheritance
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: keep inherited resource paths anchored to the original declaration.
|
||||
|
||||
Inspect `document_paths.go`, `document_paths_test.go`, the original-path map in
|
||||
`pipeline_document.go`, and sequence mutation recording in
|
||||
`document_normalizer.go`. Consult ConfigKit's public `InputInfo`, `Origin`, and
|
||||
`Source` contracts at v0.5.0.
|
||||
|
||||
Work:
|
||||
|
||||
1. Extend real-file tests using `t.TempDir()`: import parent and child pipeline
|
||||
definitions from different subdirectories beneath one configuration root.
|
||||
Give the parent pipeline a relative external reference and one reference
|
||||
inside an inherited ordered step. Assert both inherited paths resolve from
|
||||
the parent's file; a direct child reference replacement resolves from the
|
||||
child's file. Reference files may be minimal public fixture data.
|
||||
2. Add an overlay replacement case where the parent reference is declared by
|
||||
the overlay and subsequently inherited. Its base must be the overlay's
|
||||
directory. A nested reference-mapping inheritance case must preserve its
|
||||
original contributor when copied into a derived pipeline.
|
||||
3. Combine one inherited path case with whitespace-normalized pipeline or
|
||||
reference keys. The existing `originalPath` mechanism must still find the
|
||||
input's origin. ConfigKit exact sibling matching happens before Notarius
|
||||
trims names; do not pre-normalize the inheritance lookup.
|
||||
4. Include a legitimate `../` reference, retaining the current host-path
|
||||
behavior. Reuse existing coverage for built-in destination bases, explicit
|
||||
CLI destination bases, absolute paths, and empty cache roots unless the
|
||||
upgrade reveals a distinct regression.
|
||||
5. Fix `documentPathBase` or related origin handling only if these tests expose
|
||||
a failure. Query the surviving value's contributor, not the `extends`
|
||||
control location or an arbitrary sibling. Keep the existing host-path
|
||||
resolution model; do not replace it with resource opening through
|
||||
`ResourceContext.FS`, which could introduce containment restrictions.
|
||||
6. Clarify inherited declaring-file behavior in `docs/config.md`. Update the
|
||||
loading/path explanation in `docs/internal/configuration.md` to reflect
|
||||
expansion before decoding and source-aware recorded normalization.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Imported, nested, and overlaid inherited paths use the file supplying the
|
||||
surviving value; direct child replacements use their own declaring file.
|
||||
- Normalized identifiers and inherited ordered-step references retain correct
|
||||
bases without unrecorded sequence mutation failures.
|
||||
- Existing path access and destination policies are preserved. No new path
|
||||
containment, existence, or resource-format restriction is introduced.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/core/config
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Prompt size: one path-resolution seam and focused real-file regressions.
|
||||
Inspection rendering is checked separately in Stage 5.
|
||||
|
||||
## Stage 5 — Preserve Inspection, Derivation, And Ownership
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: prove inherited configuration remains correctly inspectable after
|
||||
Notarius's recorded normalization and runtime conversion.
|
||||
|
||||
Inspect `loaded_config.go`, `loaded_config_test.go`, `document_normalizer.go`,
|
||||
`redaction_test.go`, and the new inherited path fixtures. Review upstream
|
||||
metadata/history APIs rather than inventing a new provenance format.
|
||||
|
||||
Work:
|
||||
|
||||
1. Through `LoadedConfig.WriteEffective`, check a derived pipeline's expanded
|
||||
values and resolved reference paths agree with its runtime configuration.
|
||||
Verify consumed `extends` directives are absent from the effective typed
|
||||
configuration. Assert selected values/paths, not the full YAML document.
|
||||
2. Through `WriteExplanation`, inspect one inherited scalar and one inherited
|
||||
path that Notarius normalizes. Assert useful parent/source and inheritance
|
||||
evidence remains available alongside the final normalization result.
|
||||
Use structured upstream metadata within the configuration package for
|
||||
precise provenance checks, and minimal stable fragments for rendered text.
|
||||
Do not require a new Notarius renderer or exact diagnostic wording.
|
||||
3. Add a synthetic credential-shaped option to a parent and inherit it into a
|
||||
child. Verify effective and explanation views redact its value, including
|
||||
inherited/displaced history where present. Use only fake secret strings and
|
||||
the existing sensitivity predicate; add no new credential rules.
|
||||
4. Extend the existing ownership test: mutating a child's runtime nested map
|
||||
must not mutate the parent's runtime map or the frozen inspection snapshot.
|
||||
Retain the `RuntimeConfig()` detached-copy behavior without a new cloning
|
||||
mechanism unless the test demonstrates a conversion defect.
|
||||
5. Update `docs/internal/configuration.md` to describe inherited explicit input,
|
||||
retained derivation, normalization, and the detached runtime boundary. Repair
|
||||
any affected stale loading terminology, such as calling a configuration
|
||||
overlay a profile. Link to `docs/config.md` for syntax and field rules.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Effective paths agree with runtime paths; explanations retain useful source
|
||||
and transfer evidence after normalization.
|
||||
- Existing redaction applies to inherited values and history.
|
||||
- Parent, child, returned copies, and inspection snapshots have the established
|
||||
ownership behavior. Upstream inspection types stay private to configuration.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/core/config
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Prompt size: configuration inspection and ownership, with focused tests and its
|
||||
internal documentation.
|
||||
|
||||
## Stage 6 — Exercise CLI Selection And Failure Reporting
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: make derived pipelines first-class through the existing commands and
|
||||
preserve the normal load-error path for invalid inheritance.
|
||||
|
||||
Inspect `internal/cli/configkit_commands_test.go`, `command_contract_test.go`,
|
||||
and the configuration command/load helpers. Use the existing injected catalog
|
||||
and fake collaborators; do not construct providers or make LLM calls.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add a small valid parent/child fixture. Exercise `pipelines list` and
|
||||
`config validate --pipeline child`; assert the child can be selected by its
|
||||
own ID and listing remains sorted. Reuse existing command rendering tests
|
||||
rather than duplicating every output mode.
|
||||
2. Exercise `config effective` and `config explain` for that child to prove CLI
|
||||
wiring reaches the safe upstream inspection views. Assert successful status
|
||||
and relevant inherited evidence, not a whole-output snapshot.
|
||||
3. Add representative invalid load cases for a missing parent, a cycle, and an
|
||||
invalid directive type. Keep one invalid relationship in an unselected
|
||||
pipeline to prove selecting a different valid pipeline cannot hide it.
|
||||
Assert runtime-error status, no success output, and meaningful safe
|
||||
configuration diagnostics. Reuse package-level diagnostics for more detailed
|
||||
path checks; ConfigKit owns the full malformed-directive matrix.
|
||||
4. Check exact-key behavior with a whitespace-bearing parent name: the matching
|
||||
declaration works and existing identifier normalization follows expansion.
|
||||
A mismatch should produce the upstream missing-parent failure. No automatic
|
||||
trimming, aliases, or application-specific inheritance restrictions are
|
||||
added.
|
||||
5. Change production CLI code only if these tests expose a wiring defect.
|
||||
`docs/cli.md` may gain a short link to configuration inheritance where useful,
|
||||
but commands and flags remain the same.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Derived pipelines use the ordinary selection, listing, validation, and
|
||||
inspection commands.
|
||||
- Invalid relationships reach the existing load-error reporting path, even
|
||||
when their entries are not selected for execution.
|
||||
- Read-only command tests require neither credentials nor provider work.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/cli ./internal/core/config
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Prompt size: command-boundary behavior and representative error propagation.
|
||||
|
||||
## Stage 7 — Publish And Test The Development Pipeline Example
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: give operators one complete, runnable demonstration without duplicating
|
||||
the full pipeline.
|
||||
|
||||
Inspect `examples/dnd-complete.config.yml`,
|
||||
`examples/pipelines/dnd-session.yml`, `examples/overlays/development.yml`,
|
||||
`internal/cli/example_contract_test.go`, and `maintainedExampleFiles` in
|
||||
`internal/cli/production_contract_test.go`.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add this sibling to `examples/pipelines/dnd-session.yml`:
|
||||
|
||||
```yaml
|
||||
dnd-session-development:
|
||||
extends: dnd-session
|
||||
structured_output_repair_attempts: 0
|
||||
```
|
||||
|
||||
Keep it beneath `pipelines`, beside the complete parent. Explain with a short
|
||||
comment that this development variant disables PromptKit structural repairs
|
||||
while inheriting the steps and configured producer-stage retry budgets.
|
||||
It uses the parent's existing `dnd-extraction` profile.
|
||||
2. Update the complete example's expected pipeline IDs to include the child in
|
||||
sorted order. Keep the minimal example and existing operational overlay.
|
||||
Do not add a third standalone configuration or duplicate the parent's steps.
|
||||
3. Extend maintained-example coverage to load, resolve, and materialize external
|
||||
references for both complete-example pipelines. Compare the child to the
|
||||
parent for step/lane composition and generated-reference bindings, and assert
|
||||
the selected zero structural-repair setting reaches eligible resolved
|
||||
bindings. Do not assert equal pipeline digests: their IDs/settings differ.
|
||||
4. Update `TestMaintainedExamplesValidateEffectiveProfilesOffline` to validate
|
||||
every expected pipeline in each maintained example, including the new child,
|
||||
with credentials absent. Retain the existing two-root-example inventory.
|
||||
5. Update `docs/config.md`'s maintained-example guidance to show selection of the
|
||||
variant and explain that operators may instead override `llm_profile` when
|
||||
supplying another profile. Link to the inherited fragment. Add only a short
|
||||
configuration link in `docs/consumers/dnd-pipeline.md` if useful; keep its
|
||||
complete parent workflow and artifact instructions intact.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The existing complete root imports both valid pipelines, and the derived
|
||||
pipeline contains only the relationship and its intentional override.
|
||||
- Both variants resolve and validate offline using the maintained profile,
|
||||
external references, generated bindings, and full module catalog.
|
||||
- Maintained examples and navigation accurately explain the variant without
|
||||
confusing it with the operational overlay or PromptKit profile inheritance.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/cli -run 'TestMaintained'
|
||||
GOWORK=off go test ./internal/core/config ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Prompt size: one example fragment, its existing test inventory, and focused
|
||||
operator guidance. No full live extraction run is needed.
|
||||
|
||||
## Stage 8 — Consolidate Documentation And Verify The Complete Upgrade
|
||||
|
||||
> Completion: complete.
|
||||
|
||||
Goal: finish with an internally consistent, validated upgrade ready for review.
|
||||
|
||||
Inspect all changed files, `docs/config.md`,
|
||||
`docs/internal/configuration.md`, `docs/integrations/pkg-configkit.md`,
|
||||
`docs/oss/attribution.md`, and affected CLI/consumer navigation. Compare against
|
||||
the feature roadmap's acceptance criteria.
|
||||
|
||||
Work:
|
||||
|
||||
1. Confirm canonical ownership: configuration defines syntax/precedence/path
|
||||
rules; internals explain the adapter; integration documentation identifies
|
||||
the supported upstream contract; attribution records the imported version
|
||||
and notices. Remove duplicate definitions introduced during earlier stages.
|
||||
2. Check the documentation covers the reserved key in every input mapping,
|
||||
exact sibling naming, normal validation of all definitions, replacement of
|
||||
step/validator sequences, source-relative inherited paths, explicit values,
|
||||
and cancellation. Do not describe deferred `LoadLayered`, filesystem source,
|
||||
mapping source, or underlay workflows as available Notarius features.
|
||||
3. Search for stale ConfigKit v0.4.2/current-pin claims in active documentation
|
||||
and dependency files, ordinary reserved-key collisions, and confusing profile
|
||||
terminology. Leave historical releases and archived roadmaps intact. Verify
|
||||
relative document links locally and pinned external links where accessible;
|
||||
no new documentation-checking tool or recurring network gate is required.
|
||||
4. Run the full offline default suite, vet, and native build. Build to a
|
||||
`mktemp -d` destination so no binary is left in the repository. Run
|
||||
`GOWORK=off go mod tidy -diff`; an unexpected diff must be resolved and
|
||||
reviewed rather than accepted silently.
|
||||
5. Review the final implementation for unintended restrictions or guarantees:
|
||||
no new schema version, inheritance whitelist, abstract-template exemption,
|
||||
custom list merge, host-path restriction, configuration source tier,
|
||||
additional CLI knobs, or upstream types leaking into runtime contracts.
|
||||
6. Report the implemented behavior, successful checks, and any concrete gaps.
|
||||
Keep roadmap retirement and release publication for a later request.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Every required outcome in `configkit.md` is implemented and covered at the
|
||||
appropriate boundary, with no unexplained scope expansion.
|
||||
- Maintained examples and canonical documents agree with the selected module
|
||||
and observed loading behavior. Dependency metadata is tidy.
|
||||
- Full offline tests, vet, build, and relevant diff/link checks pass. Any
|
||||
unavailable optional external-link verification is reported accurately.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./...
|
||||
GOWORK=off go vet ./...
|
||||
build_dir=$(mktemp -d)
|
||||
GOWORK=off go build -o "$build_dir/notarius" ./cmd/notarius
|
||||
GOWORK=off go mod tidy -diff
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
These are development checks for this upgrade, not additional permanent release
|
||||
gates. Do not repeat successful checks for unchanged content solely to satisfy
|
||||
another handoff step; rerun the affected checks after any correction.
|
||||
|
||||
Prompt size: final review, small documentation consolidation, and aggregate
|
||||
verification. A newly discovered substantial defect must be reported with its
|
||||
evidence and required follow-up rather than folded into an unrelated refactor.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None at this baseline. The dependency target, native inheritance semantics,
|
||||
path policy, example override, documentation owners, and deferred features are
|
||||
specified. Whether an adapter needs a small correction is determined by the
|
||||
integration tests above, rather than a new product-policy decision.
|
||||
409
docs/roadmap/promptkit.md
Normal file
409
docs/roadmap/promptkit.md
Normal file
@@ -0,0 +1,409 @@
|
||||
# PromptKit 0.12 Configuration Migration
|
||||
|
||||
## Purpose
|
||||
|
||||
Adopt PromptKit v0.12.0 after the completed ConfigKit upgrade, replacing the
|
||||
removed configuration APIs and asset formats with PromptKit's ConfigKit-backed
|
||||
configuration boundary. Make the new composition, inheritance, presence, and
|
||||
captured-configuration capabilities useful without introducing a second loader
|
||||
or changing Notarius's domain extraction and pipeline acceptance policies.
|
||||
|
||||
This is a feature roadmap, not an execution sequence. The operator-source and
|
||||
native-overlay decisions were accepted on 2026-10-04 and are incorporated below.
|
||||
A separate implementation plan will define the execution sequence. Release
|
||||
preparation and publication are separate work.
|
||||
|
||||
## Reviewed Baseline
|
||||
|
||||
Reviewed on 2026-10-04 against Notarius commit
|
||||
`af62b3b`, with ConfigKit v0.5.0 and PromptKit v0.11.2 selected in `go.mod`.
|
||||
The remote PromptKit v0.12.0 tag resolves to full source commit
|
||||
`4b8169da286917fde7875425bb32b799b12c6af3`, matching the sibling checkout.
|
||||
Primary upstream references are the
|
||||
[release guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/releases/v0.12.0.md),
|
||||
[consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api),
|
||||
and [format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md).
|
||||
These were reviewed in the local checkout; the remote tag was verified directly.
|
||||
|
||||
The current integration has these migration-sensitive boundaries:
|
||||
|
||||
- `internal/framework/llm/asset_registry.go` flattens module-owned prompt,
|
||||
schema, and fallback-profile filesystems and constructs `WithPromptFS`,
|
||||
`WithSchemaFS`, and `WithFallbackProfileFS` options.
|
||||
- `promptkit_profiles.go` shares profile-source construction between the
|
||||
execution client and credential-free explicit-profile inspection. It uses
|
||||
`Config.ProfileDir`, `WithProfileFile`, and the backend registration option.
|
||||
The inspection engine supplies `PromptDir: "."` without preparing a prompt.
|
||||
- `promptkit_client.go` constructs the engine and uses prepared execution,
|
||||
semantic correction messages, structural repair overrides, observations,
|
||||
direct sessions, reasoning overrides, and provider-neutral error adaptation.
|
||||
- `promptkit_profile_fingerprint.go` combines a pinned catalog marker with
|
||||
reread operator profile files and fallback asset bytes. The optional local
|
||||
endpoint contributes a separate identity; local concurrency is excluded.
|
||||
- Embedded prompt definitions throughout `assets/dnd/` and
|
||||
`assets/generic/normalize/deduplication/` use top-level `id` and `version`.
|
||||
`assets/dnd/profiles/dnd-extraction.yaml` and the maintained deployment file
|
||||
`examples/llm-profiles/dnd-extraction.yml` use top-level `id`.
|
||||
- The public profile contract says a matching operator profile completely
|
||||
replaces a fallback or catalog profile. It documents `base_profile`
|
||||
inheritance. These claims do not match v0.12.0's native semantics.
|
||||
|
||||
The current focused LLM, CLI, and module-integration tests pass with
|
||||
`GOWORK=off`. This establishes the baseline only: no trial dependency upgrade,
|
||||
live inference, or production model-quality evaluation has been performed.
|
||||
|
||||
## Upstream Changes Relevant To Notarius
|
||||
|
||||
PromptKit v0.12.0 is an incompatible configuration/API release:
|
||||
|
||||
- `Config.Sources` accepts ordered ConfigKit underlays and an optional main
|
||||
input. Legacy prompt/profile directory fields and configuration-source
|
||||
options are removed. Schema and injected-runtime options remain available.
|
||||
- Documents use keyed `backends`, `profiles`, and `prompts` mappings, with a
|
||||
second version mapping beneath each prompt ID. Embedded `id`, `version`, and
|
||||
`base_profile` fields are invalid; profile inheritance uses `extends`.
|
||||
- Files within one source group compose strictly, not by filename precedence.
|
||||
Later groups overlay earlier groups recursively. Explicit null, zero,
|
||||
false, and empty values can differ from omission; `{}` does not clear a map.
|
||||
- Engine construction validates every final effective definition, including
|
||||
unused definitions, and captures message content and static templates.
|
||||
Ordinary execution and inspection do not reload them. Schemas, request
|
||||
artifacts, credentials, and provider operations retain their runtime timing.
|
||||
- `InspectConfiguration` exposes captured presence, sources, diagnostics,
|
||||
explanations, and redacted effective output. These reports are not
|
||||
automatically safe to publish: ordinary prompt/provider content remains data.
|
||||
- Maintained catalogs move to `/v3`, both at `v3.20260930.0`.
|
||||
PromptKit and Notarius already share the required ConfigKit v0.5.0 and Go
|
||||
1.25.5 baseline; no additional Go-version upgrade is required by this release.
|
||||
|
||||
## Target End State
|
||||
|
||||
### One Native Configuration Boundary
|
||||
|
||||
PromptKit owns the effective backend/profile/prompt configuration and all
|
||||
upstream composition and inheritance. Notarius selects its sources and adapts
|
||||
the resulting engine to the existing provider-neutral completion contract.
|
||||
ConfigKit and PromptKit types remain inside the LLM adapter's explicit
|
||||
integration boundary, not in domain modules or pipeline contracts.
|
||||
|
||||
Retain Notarius configuration `version: 5`, `promptkit.profile_file`,
|
||||
`promptkit.profile_dir`, their mutual exclusion and declaring-file-relative
|
||||
path handling, `transient_retries`, and the optional conventional local backend.
|
||||
Do not add replacement Notarius configuration fields merely because the
|
||||
upstream constructor has changed. Document the changed external file format.
|
||||
|
||||
Use one shared source-construction path for execution and profile preflight.
|
||||
The source precedence is:
|
||||
|
||||
1. PromptKit's maintained catalog layer, supplied by PromptKit itself.
|
||||
2. Registered application prompt and fallback-profile assets.
|
||||
3. The explicitly selected operator file or directory.
|
||||
4. A programmatic `local` backend contribution when Notarius's
|
||||
`promptkit.local_backend` is configured, so those explicit Notarius values
|
||||
win over conflicting file declarations of those fields.
|
||||
|
||||
The existing selected file/directory accepts native combined `profiles`,
|
||||
`backends`, and `prompts` documents. Enable explicit ConfigKit imports on these
|
||||
operator inputs. Describe the familiar `profile_file` and `profile_dir` fields
|
||||
as PromptKit configuration sources whose principal workflow is profile
|
||||
selection; do not add a section whitelist or a custom merge language.
|
||||
Domain modules remain responsible for interpreting and validating model output,
|
||||
even when an operator supplies a prompt overlay.
|
||||
|
||||
Inputs within each group use native strict composition. Directory discovery is
|
||||
recursive over YAML files and has no hidden per-file priority. No unselected
|
||||
directory is scanned. Construct filesystem-backed inputs with usable resource
|
||||
contexts so referenced message files retain their declaring-source base.
|
||||
Do not combine host profile files with embedded content under a fabricated
|
||||
common filesystem root.
|
||||
|
||||
Document import layouts that do not accidentally select the same definitions
|
||||
through both directory discovery and imports: a file-root import layout is
|
||||
often the simplest choice. Use native import/resource rules rather than
|
||||
inventing deduplication, filename override order, or broader host access.
|
||||
|
||||
Use `FilesystemInputs` for the existing flattened embedded views,
|
||||
`DirectoryInputs` or an individual ConfigKit file input for the operator
|
||||
source, and `BackendSource` with the presence-aware local-backend helper for
|
||||
programmatic backend configuration. Keep schemas on the existing schema
|
||||
filesystem boundary. There is no need to deserialize catalog assets, register
|
||||
catalog backends individually, or recreate PromptKit inheritance.
|
||||
|
||||
Profile preflight does not need dummy prompt directories or synthetic prompts.
|
||||
Both construction paths include the same catalog, application prompt/fallback,
|
||||
operator, and local-backend layers. This matters for partial operator prompt
|
||||
overlays: inspecting a layer without its application base would validate a
|
||||
different effective definition. Call `InspectProfile` on the constructed engine
|
||||
without preparing a synthetic request. Construction still captures and
|
||||
validates the effective prompt definitions; preflight does not resolve
|
||||
credential values, prepare request artifacts, or contact providers.
|
||||
|
||||
### Keyed Assets Without Prompt Behavior Changes
|
||||
|
||||
Every maintained prompt declaration becomes a keyed document of this shape:
|
||||
|
||||
```yaml
|
||||
prompts:
|
||||
dnd.spells:
|
||||
v1:
|
||||
# Existing definition body, without embedded id/version fields.
|
||||
```
|
||||
|
||||
Profiles become keyed entries beneath `profiles`. Preserve literal prompt IDs,
|
||||
the `v1` version, logical profile IDs, selected response schemas, content paths,
|
||||
message order, roles, shared fragment bytes, cache-control metadata, and
|
||||
configured structural repair budgets. Do not conflate this migration with
|
||||
prompt rewriting, schema changes, or durable artifact version changes.
|
||||
|
||||
The application-owned `dnd-extraction` fallback continues to explicitly select
|
||||
`openrouter`, `openai/gpt-5.6-luna`, timeout 240, and service tier `flex`, with
|
||||
reasoning effort unset. Do not replace that choice with an upstream catalog
|
||||
alias or adopt unrelated model defaults. Migrate the maintained operator
|
||||
example and relevant inline Go/YAML test fixtures as well as production assets.
|
||||
|
||||
Module ownership, ordered asset manifests, scoped mounting, and selected-asset
|
||||
hashing remain intact. Check virtual filenames and source-relative
|
||||
`content_file` resolution after wrapping definitions; do not move business
|
||||
logic into the root `assets` package.
|
||||
|
||||
### Presence, Inheritance, And Operator Migration
|
||||
|
||||
The adopted policy is native recursive overlay, not a compatibility shim
|
||||
for complete-definition replacement. Operators can override only the fields
|
||||
they want. Omitted fields retain lower-layer values, and `extends` uses native
|
||||
literal sibling-key lookup. Changing a backend or model does not erase
|
||||
explicit inherited settings; backend-derived defaults are applied to the
|
||||
final backend after inheritance.
|
||||
|
||||
Document intentional clearing with `null`, including reasoning/service tier,
|
||||
sampling controls, credential lookup, and provider maps where supported.
|
||||
Distinguish null from zero/false/blank and from an empty mapping. For example,
|
||||
switching an operator's `dnd-extraction` overlay to a local model may also need
|
||||
`service_tier: null` to remove the fallback's explicit `flex` setting. Do not
|
||||
automatically scrub settings based on model-name heuristics.
|
||||
|
||||
Translate `base_profile` to `extends` in maintained examples and explain that
|
||||
operators must migrate their external files. Do not retain legacy aliases or
|
||||
an alternate parser. Preserve exact PromptKit keys; do not introduce a new
|
||||
identifier normalization pass. Update tests that assumed whole replacement
|
||||
or selection-time validation deliberately, rather than treating those
|
||||
intentional upstream changes as regressions.
|
||||
|
||||
Native eager validation applies to final effective values, not independently
|
||||
to each superseded contribution. Do not add a source-history audit or raw-layer
|
||||
validation that prevents legitimate overrides from repairing lower layers.
|
||||
|
||||
### Native Operator Capabilities
|
||||
|
||||
Support the capabilities of the unified format directly, without new Notarius
|
||||
configuration knobs or model-specific adaptation:
|
||||
|
||||
- Custom or multiple backend definitions, including endpoint, credential
|
||||
environment name, provider parameters, concurrency, and queue capacity.
|
||||
Operators may also overlay maintained backend settings, for example changing
|
||||
OpenRouter's admission limit. Backend IDs are configuration keys, not a
|
||||
closed registration list. Notarius's global scheduler still applies.
|
||||
- Profile inheritance, partial overrides, and recursive provider-parameter
|
||||
composition. Explicit null suppression and zero/false values follow native
|
||||
field rules. Native sibling inheritance is not limited by a Notarius
|
||||
profiles-only whitelist; prompt/backend mappings follow upstream rules too.
|
||||
- File-configured `api_key_required`, with PromptKit's exact credential
|
||||
precedence and runtime checks. In particular, true suppresses a backend's
|
||||
implicit environment source; an explicitly configured profile environment
|
||||
source remains usable. This is an operator choice, not a new mandatory
|
||||
credential policy or a reason to require credentials during preflight.
|
||||
- Deployment prompt overlays and declaring-source-relative message files,
|
||||
including imported content. Prompt version selection remains caller-owned;
|
||||
this upgrade does not infer a new module version from file order or add
|
||||
automatic latest-version selection.
|
||||
|
||||
The existing `local_backend` object remains a convenient highest-priority
|
||||
contribution for that conventional backend. Other backend settings remain
|
||||
available through the unified operator source; do not copy every upstream field
|
||||
into a parallel Notarius backend schema.
|
||||
|
||||
Keep the two maintained top-level configuration examples. Use the complete
|
||||
example's existing PromptKit deployment file and small canonical documentation
|
||||
snippets to demonstrate native inheritance/partial overrides, custom backend
|
||||
configuration, imports, null clearing, and prompt editing without adding a
|
||||
large unrelated example suite. Not every capability needs its own runnable
|
||||
pipeline.
|
||||
|
||||
### Captured Execution And Checkpoint Identity
|
||||
|
||||
One engine represents one captured configuration for its lifetime. Editing
|
||||
operator definitions or message files requires constructing a new engine;
|
||||
there is no watcher or automatic hot reload in this scope. Preserve the normal
|
||||
CLI lifecycle without promising that independently constructed preflight and
|
||||
execution engines are one transactional filesystem snapshot.
|
||||
|
||||
Compute and retain checkpoint identity from the configuration actually
|
||||
captured by the execution engine. Do not identify an old captured engine by
|
||||
files reread after it was constructed. Identity must account for effective
|
||||
profile/backend semantics and the pinned PromptKit/catalog revisions. It must
|
||||
also account for operator prompt definitions and their
|
||||
captured prompt content, not just `content_file` path strings.
|
||||
|
||||
Use public captured inspection and prompt-definition equality values where
|
||||
appropriate; do not reach into PromptKit internals. `InspectPrompt.PromptHash`
|
||||
includes captured content and is an opaque equality value. Inspection reports
|
||||
alone do not embed referenced message-file bodies. Preserve application
|
||||
module/schema fingerprints, actual-used profile provenance, and the existing
|
||||
exclusion of concurrency-only local settings from semantic checkpoint identity.
|
||||
A conservative identity across effective profiles is acceptable; per-selected-
|
||||
profile optimization is not a prerequisite for this upgrade.
|
||||
|
||||
Raw sources, configuration history, prompt bodies, endpoints, and credential
|
||||
values must not be added to ordinary manifests or checkpoint metadata. The
|
||||
new inspection API supports both private captured identity and the explicitly
|
||||
requested operator reports below. Do not automatically dump reports into run
|
||||
output or debug bundles. Hash-only identity is distinct from publishing an
|
||||
inspection report, and redaction is not a substitute for covering execution
|
||||
semantics.
|
||||
|
||||
### Operator-Facing PromptKit Inspection
|
||||
|
||||
Provide explicit CLI operations for three complementary views:
|
||||
|
||||
- **Effective configuration:** write the captured, ConfigKit-redacted PromptKit
|
||||
configuration, including effective keyed backends, profiles, and prompts.
|
||||
Use `Engine.InspectConfiguration().WriteEffective` rather than reconstructing
|
||||
YAML from Notarius's profile declarations or runtime manifests.
|
||||
- **Setting explanation:** explain one canonical JSON Pointer in that captured
|
||||
configuration, including source history, layer overrides, and inheritance.
|
||||
Use the captured inspection's explanation API. Literal IDs containing `/`
|
||||
or `~` use ordinary JSON Pointer escaping, not a dot-path interpretation.
|
||||
- **Resolved profile:** inspect one explicit profile ID with `InspectProfile`
|
||||
and present its backend-completed execution settings, including target,
|
||||
effective reasoning/provider controls, credential environment name and
|
||||
requirement, and selected backend identity. Endpoint-only profiles remain
|
||||
identifiable as such. Do not conflate configured profile fields with values
|
||||
derived from backend defaults.
|
||||
|
||||
Use the same selected Notarius configuration/overlay, registered application
|
||||
assets, operator sources, and local-backend contribution as execution and
|
||||
preflight. A report describes the engine captured for that invocation, not a
|
||||
live view of a separate running process. Constructing the engine performs
|
||||
native definition/content validation, but these commands do not need transcript
|
||||
input, execute pipeline stages, prepare requests, resolve credential values,
|
||||
admit provider capacity, or contact a model. Required-credential profiles remain
|
||||
inspectable when the named environment variable is unset.
|
||||
|
||||
Keep existing Notarius `config effective` and `config explain` behavior intact;
|
||||
the PromptKit view must be an explicit selection, not a replacement or an
|
||||
automatic second document appended to their output. Define the concrete command
|
||||
syntax in the implementation plan and maintain it in `docs/cli.md` when it
|
||||
lands. Standard output contains the selected report only; ordinary diagnostics
|
||||
and failures follow the existing CLI conventions. Unknown paths/profiles and
|
||||
invalid pointers must be distinguishable without leaking raw configuration
|
||||
through error messages.
|
||||
|
||||
Reports are operator-requested application data. Preserve upstream credential
|
||||
redaction and Notarius's existing secret-handling boundary; include credential
|
||||
variable names, never their values. Configuration explanations can contain
|
||||
overridden values and inline prompt/provider content, and effective YAML is not
|
||||
a transcript-free or generally sanitized report. Document these properties and
|
||||
operator-owned output handling without adding permission enforcement, new
|
||||
confirmation gates, automatic report persistence, or blanket content removal.
|
||||
|
||||
Effective configuration, backend-completed profile settings, and prepared
|
||||
request details are different views. These commands do not render a prompt
|
||||
with transcript inputs or return captured message-file bodies merely because
|
||||
they appear in the configuration as `content_file` references. Keep complete
|
||||
request/response inspection on the existing explicitly requested debug path.
|
||||
|
||||
### Existing Execution Policies Remain Unchanged
|
||||
|
||||
Preserve prepared execution, the exact latest-response semantic correction
|
||||
pair, prompt-prefix caching, direct sessions, reasoning overrides, cancellation,
|
||||
the shared scheduler, local/backend admission, observer accounting, error
|
||||
classification, and credential redaction. Structural repairs, transient HTTP
|
||||
retries, outer producer attempts, and validator execution budgets remain
|
||||
separate. The existing one-repair maintained prompt default remains unchanged.
|
||||
|
||||
The migration does not change extraction/normalization semantics, warning
|
||||
classification, artifact contracts, subprocess receipts, or reusable-state
|
||||
eligibility. Construction errors may occur earlier under native eager
|
||||
validation; adapt them through the existing safe adapter/CLI error boundary,
|
||||
not by exposing arbitrary upstream configuration content.
|
||||
|
||||
## Required Work
|
||||
|
||||
- Update `go.mod` and `go.sum` to PromptKit v0.12.0 and its selected `/v3`
|
||||
catalogs. Keep ConfigKit v0.5.0 and unrelated dependencies unless module
|
||||
resolution requires a change. Remove obsolete `/v2` requirements after tidy.
|
||||
- Replace removed constructor/source/backend APIs in the LLM adapter and
|
||||
shared profile-preflight construction; retain schema and injected-client
|
||||
options. Remove obsolete adapter helpers rather than layering a parallel
|
||||
configuration system beside them.
|
||||
- Migrate every registered prompt and fallback profile, the maintained
|
||||
deployment profile, and affected test fixtures to keyed definitions.
|
||||
- Adopt the unified source policy, explicit operator imports, presence/null
|
||||
behavior, native operator capabilities, and captured
|
||||
lifetime; replace live-file checkpoint fingerprinting with captured identity.
|
||||
- Add explicit effective-configuration, setting-explanation, and resolved-
|
||||
profile CLI inspection using the shared captured-engine construction and
|
||||
public PromptKit inspection APIs. Keep upstream inspection values within
|
||||
the adapter boundary and preserve ordinary Notarius configuration reports.
|
||||
- Update behavioral tests at the adapter, CLI preflight, asset registry, and
|
||||
representative assembled-module boundaries. Reuse real ConfigKit/PromptKit
|
||||
loading and offline injected clients or local HTTP fixtures.
|
||||
- Update canonical documentation when implementation lands:
|
||||
`docs/config.md` owns selected sources, precedence, profile syntax and
|
||||
migration; `docs/operations.md` owns deployment and reload guidance;
|
||||
`docs/internal/llm.md` owns construction, capture, and checkpoint mechanics;
|
||||
`docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and
|
||||
pinned links; `docs/cli.md` owns the new inspection operations and output/error
|
||||
behavior. Add a short troubleshooting workflow in `docs/operations.md` that
|
||||
links to those references. Revise consumer examples only where affected.
|
||||
- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs,
|
||||
checking their actual notices and exact source commits. PromptKit remains
|
||||
LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse
|
||||
unchanged standard texts and preserve the project's own license grant.
|
||||
|
||||
Native layered overrides change an existing documented operator contract.
|
||||
Add a concise Notarius ADR recording the accepted native composition and
|
||||
unified-source ownership/compatibility decision rather than duplicating the
|
||||
upstream manual. Existing historical ADRs need not be rewritten.
|
||||
|
||||
## Verification And Acceptance
|
||||
|
||||
Acceptance is behavioral, not an inventory of private helper names:
|
||||
|
||||
- All registered D&D and generic prompts construct and prepare with migrated
|
||||
assets, resolving shared/module content in the correct source context.
|
||||
Request identities, message content/order, schemas, and cache controls retain
|
||||
their intended behavior. Do not add exact shared-prefix-length detector tests.
|
||||
- Operator overrides, inheritance, explicit zero/null, duplicate definitions
|
||||
within a group, local backend precedence, endpoint-only profiles, and invalid
|
||||
unused final definitions follow the native configuration policy. Cover
|
||||
representative imported combined definitions, custom/backend catalog
|
||||
overlays, partial prompt overlays with their embedded base, and optional
|
||||
versus explicitly required credentials. Credential-free preflight and
|
||||
execution resolve the same source ordering and application definitions.
|
||||
- Engine/file mutation tests demonstrate captured settings and content;
|
||||
fingerprints stay tied to that engine and change when a new engine captures
|
||||
different execution-relevant settings or operator prompt content.
|
||||
- CLI inspection reports effective values, setting origins, and backend-
|
||||
completed profiles from the same source policy without transcript input,
|
||||
credentials, or provider calls. Cover representative inheritance/null
|
||||
overrides, escaped literal IDs, required-credential inspection with an unset
|
||||
variable, and invalid/unknown selectors. Verify redaction and unchanged
|
||||
ordinary Notarius reports without pinning incidental report formatting.
|
||||
- Structural repair, semantic feedback, sessions/reasoning, capacity failure,
|
||||
transient errors, safe diagnostics, and observations retain existing
|
||||
provider-neutral behavior using deterministic boundary tests.
|
||||
- Maintained minimal, complete, and inherited development pipelines validate
|
||||
without credentials or provider calls. External legacy profile files receive
|
||||
clear migration guidance, not silent reinterpretation.
|
||||
|
||||
Use focused LLM, CLI, and module-integration tests during implementation, then
|
||||
`GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, a temporary native build,
|
||||
and `GOWORK=off go mod tidy -diff` for this shared-boundary migration. Verify
|
||||
selected module versions, attribution links, maintained examples, and stale
|
||||
API/format claims. These are migration checks, not new permanent release gates.
|
||||
Live provider testing is optional follow-up evidence, not required acceptance.
|
||||
|
||||
This work warrants a minor Notarius release because the accepted external
|
||||
profile format and override behavior change. Keep Notarius configuration at
|
||||
version 5; select the actual release number during separate release preparation.
|
||||
108
docs/roadmap/remediation.md
Normal file
108
docs/roadmap/remediation.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# ConfigKit Inheritance Review Remediation
|
||||
|
||||
## Review Scope And Overall Result
|
||||
|
||||
Reviewed on 2026-10-04, comparing implementation commits `ec99d48` through
|
||||
`6450994` with the baseline `9e74df0`, the ConfigKit feature roadmap, and its
|
||||
eight-stage implementation plan.
|
||||
|
||||
The architecture matches the intended design. ConfigKit v0.5.0 owns source
|
||||
composition and native sibling inheritance before decoding. Notarius retains
|
||||
its typed defaults, presence-sensitive validation, declaration-relative path
|
||||
normalization, detached runtime configuration, catalog-dependent resolution,
|
||||
and private redacted inspection boundary. The production configuration adapter
|
||||
needed no changes; the work adds the dependency update, integration tests,
|
||||
documentation, and the derived development pipeline example.
|
||||
|
||||
No production regression or inadvertently strengthened requirement was found.
|
||||
In particular, the implementation adds no inheritance whitelist, new schema
|
||||
version, custom merging, path-containment rule, template exemption, or release
|
||||
gate. The newly reserved `extends` key is the expressly adopted upstream
|
||||
compatibility change.
|
||||
|
||||
Two follow-ups were identified: a path-test blind spot and an incomplete
|
||||
documented command. Completion is recorded under each finding. This document
|
||||
records remediation work only; it does not authorize
|
||||
unrelated runtime refactoring.
|
||||
|
||||
## Finding 1 — Make The Path Fixture Distinguish Declaring Directories
|
||||
|
||||
Priority: P2. Category: verification gap.
|
||||
|
||||
Status: addressed on 2026-10-04. The fixture now uses distinct parent-, child-,
|
||||
and overlay-local paths for the source-base assertions and retains a separate
|
||||
`../` reference. `GOWORK=off go test ./internal/core/config` passed. No
|
||||
production resolver change was needed.
|
||||
|
||||
Evidence: `internal/core/config/inheritance_test.go:338–436`,
|
||||
`TestInheritedReferencePathsKeepDeclaringSource`.
|
||||
|
||||
The parent, child, and overlay files live in `root/parents`, `root/children`,
|
||||
and `root/overlays`. Their resource paths all use `../fixtures/<name>`, and
|
||||
the expected paths all point into `root/fixtures`. Resolving one of these
|
||||
paths from any of the three directories yields the same result. Thus, choosing
|
||||
the child's directory for an inherited step/nested reference, or choosing the
|
||||
parent's directory for an overlay replacement, could still pass this test.
|
||||
|
||||
`TestInheritedInspectionRetainsOriginsRedactionAndOwnership` does distinguish
|
||||
the parent directory for a simple inherited `./shared.txt` reference. It does
|
||||
not close the blind spot for the nested, ordered-step, direct-child replacement,
|
||||
and overlay cases in the larger path fixture.
|
||||
|
||||
Required change:
|
||||
|
||||
- Revise the existing path fixture to use directory-distinguishing relative
|
||||
paths for these cases, such as `./shared.txt`, `./step.txt`, and a child-local
|
||||
replacement. Assert inherited paths against the parent's directory, direct
|
||||
replacements against the child's directory, and overlay-supplied paths
|
||||
against the overlay's directory.
|
||||
- Retain a separate legitimate `../` example to protect the existing host-path
|
||||
behavior. Avoid making every path converge on one shared sibling directory.
|
||||
- Keep this as a focused real-file configuration test. Change production
|
||||
resolution only if the improved assertions expose an actual defect.
|
||||
|
||||
Acceptance: choosing a parent, child, or overlay directory incorrectly produces
|
||||
a different absolute path and fails the relevant assertion. Nested inheritance,
|
||||
inherited ordered-step references, child replacements, and overlay replacements
|
||||
still pass with the actual resolver.
|
||||
|
||||
Validation: `GOWORK=off go test ./internal/core/config`.
|
||||
|
||||
## Finding 2 — Supply Input In The Development Pipeline Command
|
||||
|
||||
Priority: P3. Category: documentation accuracy.
|
||||
|
||||
Status: addressed on 2026-10-04. The example now supplies the pipeline ID,
|
||||
configuration path, and transcript input path, and links to the CLI run
|
||||
reference. The flags were checked against that reference, and
|
||||
`GOWORK=off go test ./internal/cli` passed without a live model request.
|
||||
|
||||
Evidence: `docs/config.md:43–49`, in Maintained Examples.
|
||||
|
||||
The new example says to select the development pipeline with
|
||||
`notarius run dnd-session-development --config /path/to/dnd-complete.config.yml`.
|
||||
The run command requires `--input`. Executing the equivalent command against
|
||||
the maintained configuration returns `notarius: run requires --input` with CLI
|
||||
exit status 2, before pipeline execution.
|
||||
|
||||
Required change: include a transcript input placeholder in the invocation,
|
||||
for example `--input /path/to/transcript.json`, and retain the link to the CLI
|
||||
reference for the full invocation contract. No CLI behavior change or new test
|
||||
framework is needed.
|
||||
|
||||
Acceptance: the documented command supplies both required run inputs—the
|
||||
pipeline ID and input path—and uses the implemented flag names. Verify against
|
||||
the CLI reference and existing parser coverage without a live model request.
|
||||
|
||||
## Checks Completed During Review
|
||||
|
||||
- `GOWORK=off go test ./...` passed.
|
||||
- `GOWORK=off go vet ./...` passed.
|
||||
- `GOWORK=off go build` of `./cmd/notarius` to a temporary destination passed.
|
||||
- `GOWORK=off go mod tidy -diff` produced no changes.
|
||||
- The selected ConfigKit module is v0.5.0; PromptKit remains v0.11.2.
|
||||
- The implementation diff passed whitespace checks.
|
||||
- The incomplete development invocation was reproduced without provider calls.
|
||||
|
||||
These checks describe the review baseline. Subsequent remediation and its
|
||||
focused validation are recorded under each finding.
|
||||
@@ -162,3 +162,9 @@ pipelines:
|
||||
module: appendorder
|
||||
normalize:
|
||||
module: dnd/enemy-events
|
||||
|
||||
# Development variant: disable PromptKit structural repairs while retaining
|
||||
# the inherited steps and configured producer retry budgets.
|
||||
dnd-session-development:
|
||||
extends: dnd-session
|
||||
structured_output_repair_attempts: 0
|
||||
|
||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
||||
go 1.25.5
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.2
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.5.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.2
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
|
||||
4
go.sum
4
go.sum
@@ -1,5 +1,5 @@
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.2 h1:ZMxmv8pOnLtAx2hFTf8Z+pHKUbR89xErkBhwjr4yK24=
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.2/go.mod h1:tYWpsrbLyu05r8+1MapWp5fN6ZrX9I8hU2xEPedYmYk=
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.5.0 h1:2KwG10+kApWM3avOoYA+p768xcbHUp7tQ2rqizBNpyg=
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.5.0/go.mod h1:tYWpsrbLyu05r8+1MapWp5fN6ZrX9I8hU2xEPedYmYk=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.2 h1:2RqI+KtflvSh7vLQaTAQCmMaFQscEA7EPwPazyvXRoY=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.2/go.mod h1:onm/FBPUKQ8/NSPwTEk26Ut+dVX2iXcq8vEseB+g/0E=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260926.0 h1:wdab+R+lgMZxzYwhbAuCrW7zdSQ1cW9ovBR6UzLmZR4=
|
||||
|
||||
@@ -40,6 +40,130 @@ func TestConfigKitReadOnlyCommandsLoadImportsAndOverlays(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPipelineInheritanceCommandsSelectAndInspectDerivedPipelines(t *testing.T) {
|
||||
t.Parallel()
|
||||
path := writeCommandConfigContent(t, `version: 5
|
||||
pipelines:
|
||||
" parent ":
|
||||
input:
|
||||
module: seriatim
|
||||
artifacts:
|
||||
spells:
|
||||
extract:
|
||||
module: dnd/spells
|
||||
structured_output_repair_attempts: 2
|
||||
child:
|
||||
extends: " parent "
|
||||
structured_output_repair_attempts: 0
|
||||
`)
|
||||
options := commandContractOptions(t)
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
if code := RunWithOptions([]string{"pipelines", "list", "--config", path}, &stdout, &stderr, options); code != 0 || stdout.String() != "child\nparent\n" || stderr.Len() != 0 {
|
||||
t.Fatalf("list = %d, %q, %q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
if code := RunWithOptions([]string{"config", "validate", "--config", path, "--pipeline", "child"}, &stdout, &stderr, options); code != 0 || !strings.Contains(stdout.String(), `is valid for pipeline "child"`) || stderr.Len() != 0 {
|
||||
t.Fatalf("validate child = %d, %q, %q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
if code := RunWithOptions([]string{"config", "effective", "--config", path}, &stdout, &stderr, options); code != 0 || !strings.Contains(stdout.String(), "module: seriatim") || !strings.Contains(stdout.String(), "structured_output_repair_attempts: 0") || strings.Contains(stdout.String(), "extends:") || stderr.Len() != 0 {
|
||||
t.Fatalf("effective = %d, %q, %q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
if code := RunWithOptions([]string{"config", "explain", "--config", path, "/pipelines/child/input/module"}, &stdout, &stderr, options); code != 0 || !strings.Contains(stdout.String(), `path: "/pipelines/child/input/module"`) || !strings.Contains(stdout.String(), "parent") || stderr.Len() != 0 {
|
||||
t.Fatalf("explain inherited input = %d, %q, %q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPipelineInheritanceLoadFailuresUseRuntimeDiagnostics(t *testing.T) {
|
||||
t.Parallel()
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
content string
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "missing parent",
|
||||
content: `version: 5
|
||||
pipelines:
|
||||
child:
|
||||
extends: absent
|
||||
`,
|
||||
args: []string{"config", "validate", "--pipeline", "child"},
|
||||
want: "inheritance.missing",
|
||||
},
|
||||
{
|
||||
name: "cycle",
|
||||
content: `version: 5
|
||||
pipelines:
|
||||
first:
|
||||
extends: second
|
||||
second:
|
||||
extends: first
|
||||
`,
|
||||
args: []string{"config", "validate", "--pipeline", "first"},
|
||||
want: "cycle",
|
||||
},
|
||||
{
|
||||
name: "invalid directive type",
|
||||
content: `version: 5
|
||||
pipelines:
|
||||
child:
|
||||
extends: 7
|
||||
`,
|
||||
args: []string{"config", "validate", "--pipeline", "child"},
|
||||
want: "extends",
|
||||
},
|
||||
{
|
||||
name: "unselected missing parent",
|
||||
content: `version: 5
|
||||
pipelines:
|
||||
parent:
|
||||
input:
|
||||
module: seriatim
|
||||
child:
|
||||
extends: parent
|
||||
unrelated:
|
||||
extends: absent
|
||||
`,
|
||||
args: []string{"config", "validate", "--pipeline", "child"},
|
||||
want: "inheritance.missing",
|
||||
},
|
||||
{
|
||||
name: "exact sibling mismatch",
|
||||
content: `version: 5
|
||||
pipelines:
|
||||
parent:
|
||||
input:
|
||||
module: seriatim
|
||||
child:
|
||||
extends: "parent "
|
||||
`,
|
||||
args: []string{"pipelines", "list"},
|
||||
want: "inheritance.missing",
|
||||
},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
path := writeCommandConfigContent(t, test.content)
|
||||
args := append([]string{}, test.args...)
|
||||
args = append(args, "--config", path)
|
||||
var stdout, stderr bytes.Buffer
|
||||
if code := RunWithOptions(args, &stdout, &stderr, commandContractOptions(t)); code != 1 || stdout.Len() != 0 || stderr.Len() == 0 || !strings.Contains(stderr.String(), test.want) {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetiredConfigProfileFlagUsesNormalUnknownFlagBehavior(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
@@ -25,6 +26,7 @@ import (
|
||||
|
||||
func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
||||
components := productionTestComponents(t)
|
||||
completePipelines := make(map[string]pipeline.ResolvedPipeline)
|
||||
for _, example := range maintainedExampleFiles(t) {
|
||||
t.Run(example.name, func(t *testing.T) {
|
||||
cfg := loadMaintainedExample(t, example.path)
|
||||
@@ -58,9 +60,45 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
||||
t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err)
|
||||
}
|
||||
if example.name == "complete" {
|
||||
if pipelineID == "dnd-session-development" {
|
||||
parent, found := completePipelines["dnd-session"]
|
||||
if !found {
|
||||
t.Fatal("complete parent pipeline was not resolved before its development variant")
|
||||
}
|
||||
if strings.Join(exampleStepLaneIDs(materialized), "|") != strings.Join(exampleStepLaneIDs(parent), "|") {
|
||||
t.Fatalf("development variant steps and lanes = %v, want parent composition %v", exampleStepLaneIDs(materialized), exampleStepLaneIDs(parent))
|
||||
}
|
||||
if got, want := strings.Join(generatedReferenceBindingDescriptions(materialized), "|"), strings.Join(generatedReferenceBindingDescriptions(parent), "|"); got != want {
|
||||
t.Fatalf("development generated reference bindings = %q, want parent bindings %q", got, want)
|
||||
}
|
||||
eligibleBindings := 0
|
||||
for _, step := range materialized.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
for _, binding := range []struct {
|
||||
module pipeline.ModuleBinding
|
||||
executionClass contracts.ExecutionClass
|
||||
}{
|
||||
{module: lane.Extract, executionClass: lane.ExtractExecutionClass},
|
||||
{module: lane.Normalize, executionClass: lane.NormalizeExecutionClass},
|
||||
} {
|
||||
if binding.executionClass != "llm_backed" {
|
||||
continue
|
||||
}
|
||||
eligibleBindings++
|
||||
if got := binding.module.StructuredOutputRepairAttempts; got == nil || *got != 0 {
|
||||
t.Fatalf("development %s repair attempts = %v, want explicit zero", binding.module.Module, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if eligibleBindings == 0 {
|
||||
t.Fatal("development variant had no eligible LLM bindings for repair policy")
|
||||
}
|
||||
}
|
||||
if got := exampleStepLaneIDs(materialized); strings.Join(got, "|") != "describe-session:item-registry,location-registry,npc-registry,scene-descriptions|extract-events:combat-turns,item-occurrences,location-occurrences,npc-occurrences,spells|track-enemies:enemy-events" {
|
||||
t.Fatalf("complete example steps and lanes = %v, want the documented D&D extractor composition", got)
|
||||
}
|
||||
completePipelines[pipelineID] = materialized
|
||||
locationLane := referenceContractLane(t, materialized, "location-registry")
|
||||
if locationLane.ArtifactKind != dnd.LocationRegistryKind || locationLane.Extract.Module != locationextract.Key || locationLane.Extract.Retries != 2 || locationLane.Merge.Module != pipeline.DefaultMergeModule || locationLane.Normalize.Module != locationnormalize.Key || locationLane.Normalize.Retries != 2 {
|
||||
t.Fatalf("location lane = %#v, want typed registry composition", locationLane)
|
||||
@@ -164,12 +202,16 @@ func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
|
||||
t.Setenv("OPENROUTER_API_KEY", "")
|
||||
for _, example := range maintainedExampleFiles(t) {
|
||||
t.Run(example.name, func(t *testing.T) {
|
||||
var stdout, stderr strings.Builder
|
||||
code := RunWithOptions([]string{
|
||||
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
|
||||
}, &stdout, &stderr, Options{})
|
||||
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
for _, pipelineID := range example.pipelineIDs {
|
||||
t.Run(pipelineID, func(t *testing.T) {
|
||||
var stdout, stderr strings.Builder
|
||||
code := RunWithOptions([]string{
|
||||
"config", "validate", "--config", example.path, "--pipeline", pipelineID,
|
||||
}, &stdout, &stderr, Options{})
|
||||
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "`+pipelineID+`"`) {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -188,6 +230,31 @@ func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||
return result
|
||||
}
|
||||
|
||||
func generatedReferenceBindingDescriptions(resolved pipeline.ResolvedPipeline) []string {
|
||||
var descriptions []string
|
||||
for _, step := range resolved.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
for _, target := range []struct {
|
||||
name string
|
||||
target pipeline.ResolvedReferenceTarget
|
||||
}{
|
||||
{name: "extract", target: lane.ExtractReferences},
|
||||
{name: "merge", target: lane.MergeReferences},
|
||||
{name: "normalize", target: lane.NormalizeReferences},
|
||||
} {
|
||||
for _, binding := range target.target.Bindings {
|
||||
if binding.Artifact == nil {
|
||||
continue
|
||||
}
|
||||
descriptions = append(descriptions, fmt.Sprintf("%s/%s/%s/%s=%s/%s", step.ID, lane.ID, target.name, binding.SlotName, binding.Artifact.Step, binding.Artifact.Lane))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(descriptions)
|
||||
return descriptions
|
||||
}
|
||||
|
||||
func TestMaintainedMinimalInvocationProducesJSONBundle(t *testing.T) {
|
||||
outputRoot := filepath.Join(t.TempDir(), "output")
|
||||
fake := &productionFakeLLMClient{}
|
||||
|
||||
@@ -824,7 +824,7 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
||||
t.Helper()
|
||||
return []maintainedExample{
|
||||
{name: "minimal", path: repositoryPath("examples", "dnd-minimal.config.yml"), transcriptPath: repositoryPath("examples", "seriatim-minimal-transcript.json"), pipelineIDs: []string{"dnd-session"}},
|
||||
{name: "complete", path: repositoryPath("examples", "dnd-complete.config.yml"), transcriptPath: repositoryPath("examples", "dnd-complete-transcript.json"), pipelineIDs: []string{"dnd-session"}},
|
||||
{name: "complete", path: repositoryPath("examples", "dnd-complete.config.yml"), transcriptPath: repositoryPath("examples", "dnd-complete-transcript.json"), pipelineIDs: []string{"dnd-session", "dnd-session-development"}},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
619
internal/core/config/inheritance_test.go
Normal file
619
internal/core/config/inheritance_test.go
Normal file
@@ -0,0 +1,619 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
)
|
||||
|
||||
func TestPipelineInheritancePreservesCompositionAndReplacesSteps(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
childSteps string
|
||||
wantStepIDs []string
|
||||
}{
|
||||
{
|
||||
name: "inherits ordered steps",
|
||||
wantStepIDs: []string{"prepare", "summarize"},
|
||||
},
|
||||
{
|
||||
name: "replaces ordered steps",
|
||||
childSteps: `
|
||||
steps:
|
||||
- id: replacement
|
||||
artifacts:
|
||||
summary:
|
||||
extract: {module: extract}`,
|
||||
wantStepIDs: []string{"replacement"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
source := `version: 5
|
||||
pipelines:
|
||||
parent:
|
||||
input: {module: input}
|
||||
structured_output_repair_attempts: 2
|
||||
steps:
|
||||
- id: prepare
|
||||
artifacts:
|
||||
score:
|
||||
extract: {module: extract, retries: 2}
|
||||
- id: summarize
|
||||
artifacts:
|
||||
notes:
|
||||
extract: {module: extract}
|
||||
child:
|
||||
extends: parent
|
||||
llm_profile: local
|
||||
structured_output_repair_attempts: 0` + test.childSteps + "\n"
|
||||
loaded, err := loadConfigSource(configkit.Bytes("inheritance.yml", []byte(source)), LoadOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("loadConfigSource() error = %v", err)
|
||||
}
|
||||
|
||||
parent := loaded.Config.Pipelines["parent"]
|
||||
child := loaded.Config.Pipelines["child"]
|
||||
if parent.Input.Module != "input" || child.Input.Module != "input" {
|
||||
t.Fatalf("parent/child inputs = %#v / %#v", parent.Input, child.Input)
|
||||
}
|
||||
if child.LLMProfile != "local" ||
|
||||
child.StructuredOutputRepairAttempts == nil || *child.StructuredOutputRepairAttempts != 0 {
|
||||
t.Fatalf("child overrides = %#v", child)
|
||||
}
|
||||
if parent.StructuredOutputRepairAttempts == nil || *parent.StructuredOutputRepairAttempts != 2 {
|
||||
t.Fatalf("parent settings changed: %#v", parent)
|
||||
}
|
||||
if len(child.Steps) != len(test.wantStepIDs) {
|
||||
t.Fatalf("child steps = %#v, want %v", child.Steps, test.wantStepIDs)
|
||||
}
|
||||
for i, want := range test.wantStepIDs {
|
||||
if child.Steps[i].ID != want {
|
||||
t.Fatalf("child step %d = %q, want %q", i, child.Steps[i].ID, want)
|
||||
}
|
||||
}
|
||||
if test.childSteps == "" && child.Steps[0].Artifacts["score"].Extract.Module != "extract" {
|
||||
t.Fatalf("inherited first artifact = %#v", child.Steps[0].Artifacts["score"])
|
||||
}
|
||||
|
||||
for _, id := range []string{"parent", "child"} {
|
||||
resolved, err := loaded.Config.Resolve(ResolveInput{PipelineID: id, Catalog: effectiveCatalog(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(%q) error = %v", id, err)
|
||||
}
|
||||
if resolved.ResolvedPipeline.ID != id {
|
||||
t.Fatalf("resolved pipeline ID = %q, want %q", resolved.ResolvedPipeline.ID, id)
|
||||
}
|
||||
wantResolvedSteps := len(test.wantStepIDs)
|
||||
if id == "parent" {
|
||||
wantResolvedSteps = 2
|
||||
}
|
||||
if len(resolved.ResolvedPipeline.Steps) != wantResolvedSteps {
|
||||
t.Fatalf("resolved %q steps = %#v", id, resolved.ResolvedPipeline.Steps)
|
||||
}
|
||||
if id == "child" && resolved.ResolvedPipeline.Steps[0].ArtifactLanes[0].Extract.LLMProfile != "local" {
|
||||
t.Fatalf("resolved child profile = %q", resolved.ResolvedPipeline.Steps[0].ArtifactLanes[0].Extract.LLMProfile)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactLaneInheritanceUsesSiblingMappingComposition(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
loaded, err := loadConfigSource(configkit.Bytes("lanes.yml", []byte(`version: 5
|
||||
pipelines:
|
||||
session:
|
||||
input: {module: input}
|
||||
artifacts:
|
||||
base:
|
||||
extract: {module: extract, retries: 2}
|
||||
derived:
|
||||
extends: base
|
||||
extract: {retries: 4}
|
||||
`)), LoadOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("loadConfigSource() error = %v", err)
|
||||
}
|
||||
|
||||
lanes := loaded.Config.Pipelines["session"].Artifacts
|
||||
if lanes["base"].Extract.Module != "extract" || lanes["base"].Extract.Retries != 2 {
|
||||
t.Fatalf("base lane = %#v", lanes["base"])
|
||||
}
|
||||
if lanes["derived"].Extract.Module != "extract" || lanes["derived"].Extract.Retries != 4 {
|
||||
t.Fatalf("derived lane = %#v", lanes["derived"])
|
||||
}
|
||||
if _, err := loaded.Config.Resolve(ResolveInput{PipelineID: "session", Catalog: effectiveCatalog(t)}); err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInheritedPresenceOverridesRemainExplicit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
loaded, err := loadConfigSource(configkit.Bytes("presence.yml", []byte(`version: 5
|
||||
pipelines:
|
||||
parent:
|
||||
input: {module: input}
|
||||
structured_output_repair_attempts: 3
|
||||
chunk:
|
||||
module: generic
|
||||
validators: []
|
||||
artifacts:
|
||||
score:
|
||||
extract:
|
||||
module: extract
|
||||
validators: [{module: configured-a}]
|
||||
child:
|
||||
extends: parent
|
||||
structured_output_repair_attempts: 0
|
||||
artifacts:
|
||||
score:
|
||||
extract:
|
||||
validators: []
|
||||
`)), LoadOptions{})
|
||||
if err != nil {
|
||||
var loadError *configkit.LoadError
|
||||
if errors.As(err, &loadError) {
|
||||
t.Fatalf("loadConfigSource() diagnostics = %#v", loadError.Diagnostics())
|
||||
}
|
||||
t.Fatalf("loadConfigSource() error = %v", err)
|
||||
}
|
||||
|
||||
parent := loaded.Config.Pipelines["parent"]
|
||||
child := loaded.Config.Pipelines["child"]
|
||||
if parent.Chunk.Module != "generic" || !parent.Chunk.Validators.Set || parent.Chunk.Validators.Validators == nil || len(parent.Chunk.Validators.Validators) != 0 {
|
||||
t.Fatalf("parent empty chunk validators = %#v", parent.Chunk)
|
||||
}
|
||||
if !parent.Artifacts["score"].Extract.Validators.Set || len(parent.Artifacts["score"].Extract.Validators.Validators) != 1 ||
|
||||
parent.Artifacts["score"].Extract.Validators.Validators[0].Module != "configured-a" {
|
||||
t.Fatalf("parent extraction validators = %#v", parent.Artifacts["score"].Extract.Validators)
|
||||
}
|
||||
if parent.StructuredOutputRepairAttempts == nil || *parent.StructuredOutputRepairAttempts != 3 {
|
||||
t.Fatalf("parent repair budget = %#v", parent.StructuredOutputRepairAttempts)
|
||||
}
|
||||
if child.StructuredOutputRepairAttempts == nil || *child.StructuredOutputRepairAttempts != 0 {
|
||||
t.Fatalf("child zero repair budget = %#v", child.StructuredOutputRepairAttempts)
|
||||
}
|
||||
extract := child.Artifacts["score"].Extract
|
||||
if !extract.Validators.Set || extract.Validators.Validators == nil || len(extract.Validators.Validators) != 0 {
|
||||
t.Fatalf("child empty extraction validators = %#v", extract.Validators)
|
||||
}
|
||||
if !child.Chunk.Validators.Set || child.Chunk.Validators.Validators == nil || len(child.Chunk.Validators.Validators) != 0 {
|
||||
t.Fatalf("inherited empty chunk validators = %#v", child.Chunk.Validators)
|
||||
}
|
||||
|
||||
resolved, err := loaded.Config.Resolve(ResolveInput{PipelineID: "child", Catalog: effectiveCatalog(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(child) error = %v", err)
|
||||
}
|
||||
lanes := resolved.ResolvedPipeline.AllArtifactLanes()
|
||||
if len(lanes) != 1 || lanes[0].Extract.StructuredOutputRepairAttempts == nil || *lanes[0].Extract.StructuredOutputRepairAttempts != 0 {
|
||||
t.Fatalf("resolved child repair budget = %#v", lanes)
|
||||
}
|
||||
if !resolved.ResolvedPipeline.Chunk.Validators.Set || len(resolved.ResolvedPipeline.Chunk.Validators.Validators) != 0 ||
|
||||
!lanes[0].Extract.Validators.Set || len(lanes[0].Extract.Validators.Validators) != 0 {
|
||||
t.Fatalf("resolved empty validator overrides = chunk %#v, extraction %#v", resolved.ResolvedPipeline.Chunk.Validators, lanes[0].Extract.Validators)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInheritedFalseOptionRetainsExplicitMetadata(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
loaded, err := loadConfigSource(configkit.Bytes("false-option.yml", []byte(`version: 5
|
||||
pipelines:
|
||||
parent:
|
||||
input: {module: input}
|
||||
output:
|
||||
module: json
|
||||
options:
|
||||
evidence_context: {enabled: false}
|
||||
child:
|
||||
extends: parent
|
||||
`)), LoadOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("loadConfigSource() error = %v", err)
|
||||
}
|
||||
options := loaded.Config.Pipelines["child"].Output.Options["evidence_context"].(map[string]any)
|
||||
if enabled, ok := options["enabled"].(bool); !ok || enabled {
|
||||
t.Fatalf("inherited evidence_context.enabled = %#v", options["enabled"])
|
||||
}
|
||||
path := configkit.Keys("pipelines", "child", "output", "options", "evidence_context", "enabled")
|
||||
info, ok := loaded.result.Metadata().Lookup(path)
|
||||
if !ok || !info.Explicit || info.Null || len(info.Origin.Inheritance) == 0 {
|
||||
t.Fatalf("inherited false metadata = %#v, found=%v", info, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInheritedAndDirectNullRepairBudgetsRemainInvalid(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
source string
|
||||
path string
|
||||
}{
|
||||
{
|
||||
name: "child null",
|
||||
source: `version: 5
|
||||
pipelines:
|
||||
parent: {input: {module: input}, structured_output_repair_attempts: 2}
|
||||
child:
|
||||
extends: parent
|
||||
structured_output_repair_attempts: null
|
||||
`,
|
||||
path: "/pipelines/child/structured_output_repair_attempts",
|
||||
},
|
||||
{
|
||||
name: "inherited null",
|
||||
source: `version: 5
|
||||
pipelines:
|
||||
parent: {input: {module: input}, structured_output_repair_attempts: null}
|
||||
child: {extends: parent}
|
||||
`,
|
||||
path: "/pipelines/parent/structured_output_repair_attempts",
|
||||
},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
_, err := loadConfigSource(configkit.Bytes("null.yml", []byte(test.source)), LoadOptions{})
|
||||
assertDocumentDiagnosticAt(t, err, test.path)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOverlayAndOverridesComposeBeforeInheritance(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
root := t.TempDir()
|
||||
overlays := filepath.Join(root, "overlays")
|
||||
if err := os.MkdirAll(overlays, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
main := filepath.Join(root, "config.yml")
|
||||
if err := os.WriteFile(main, []byte(`version: 5
|
||||
pipelines:
|
||||
parent:
|
||||
input: {module: input}
|
||||
structured_output_repair_attempts: 1
|
||||
inherited:
|
||||
extends: parent
|
||||
overridden:
|
||||
extends: parent
|
||||
structured_output_repair_attempts: 0
|
||||
cancelled:
|
||||
extends: parent
|
||||
input: {module: input}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(overlays, "development.yml"), []byte(`pipelines:
|
||||
parent:
|
||||
structured_output_repair_attempts: 2
|
||||
cancelled:
|
||||
extends: null
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
loaded, err := Load(main, LoadOptions{Overlay: "development"})
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if got := *loaded.Config.Pipelines["inherited"].StructuredOutputRepairAttempts; got != 2 {
|
||||
t.Fatalf("overlay-inherited repair budget = %d, want 2", got)
|
||||
}
|
||||
if got := *loaded.Config.Pipelines["overridden"].StructuredOutputRepairAttempts; got != 0 {
|
||||
t.Fatalf("child replacement repair budget = %d, want 0", got)
|
||||
}
|
||||
if loaded.Config.Pipelines["cancelled"].StructuredOutputRepairAttempts != nil {
|
||||
t.Fatalf("cancelled child inherited parent settings: %#v", loaded.Config.Pipelines["cancelled"])
|
||||
}
|
||||
|
||||
withOverride, err := Load(main, LoadOptions{
|
||||
Overlay: "development",
|
||||
Overrides: []configkit.Override{{
|
||||
Path: configkit.Keys("pipelines", "parent", "structured_output_repair_attempts"),
|
||||
Value: 3,
|
||||
Flag: "--repair-budget",
|
||||
}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Load() with explicit override error = %v", err)
|
||||
}
|
||||
if got := *withOverride.Config.Pipelines["inherited"].StructuredOutputRepairAttempts; got != 3 {
|
||||
t.Fatalf("override-inherited repair budget = %d, want 3", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInheritedReferencePathsKeepDeclaringSource(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
root := t.TempDir()
|
||||
parentDirectory := filepath.Join(root, "parents")
|
||||
childDirectory := filepath.Join(root, "children")
|
||||
overlayDirectory := filepath.Join(root, "overlays")
|
||||
fixtureDirectory := filepath.Join(root, "fixtures")
|
||||
for _, directory := range []string{parentDirectory, childDirectory, overlayDirectory, fixtureDirectory} {
|
||||
if err := os.MkdirAll(directory, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, fixture := range []struct{ directory, name string }{
|
||||
{parentDirectory, "shared.txt"},
|
||||
{parentDirectory, "parent.txt"},
|
||||
{parentDirectory, "step.txt"},
|
||||
{childDirectory, "child.txt"},
|
||||
{overlayDirectory, "overlay.txt"},
|
||||
{fixtureDirectory, "outside.txt"},
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(fixture.directory, fixture.name), []byte("fixture\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
parent := filepath.Join(parentDirectory, "pipelines.yml")
|
||||
if err := os.WriteFile(parent, []byte(`pipelines:
|
||||
" parent ":
|
||||
input: {module: input}
|
||||
references:
|
||||
" shared ": {path: ./shared.txt}
|
||||
nested:
|
||||
extends: " shared "
|
||||
source: {path: ./parent.txt}
|
||||
outside: {path: ../fixtures/outside.txt}
|
||||
steps:
|
||||
- id: first
|
||||
references:
|
||||
step-source: {path: ./step.txt}
|
||||
artifacts:
|
||||
summary:
|
||||
extract: {module: extract}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
child := filepath.Join(childDirectory, "pipelines.yml")
|
||||
if err := os.WriteFile(child, []byte(`pipelines:
|
||||
child:
|
||||
extends: " parent "
|
||||
references:
|
||||
source: {path: ./child.txt}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
main := filepath.Join(root, "notarius.yml")
|
||||
if err := os.WriteFile(main, []byte(`version: 5
|
||||
configkit:
|
||||
imports:
|
||||
- path: parents/pipelines.yml
|
||||
- path: children/pipelines.yml
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(overlayDirectory, "development.yml"), []byte(`pipelines:
|
||||
" parent ":
|
||||
references:
|
||||
" shared ": {path: ./overlay.txt}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
loaded, err := loadConfig(main, LoadOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("loadConfig() error = %v", err)
|
||||
}
|
||||
assertReferencePath := func(pipelineID, referenceID, want string) {
|
||||
t.Helper()
|
||||
if got := loaded.Config.Pipelines[pipelineID].References[referenceID].Path; got != want {
|
||||
t.Errorf("%s reference %s path = %q, want %q", pipelineID, referenceID, got, want)
|
||||
}
|
||||
}
|
||||
assertReferencePath("parent", "shared", filepath.Join(parentDirectory, "shared.txt"))
|
||||
assertReferencePath("child", "shared", filepath.Join(parentDirectory, "shared.txt"))
|
||||
assertReferencePath("child", "nested", filepath.Join(parentDirectory, "shared.txt"))
|
||||
assertReferencePath("parent", "source", filepath.Join(parentDirectory, "parent.txt"))
|
||||
assertReferencePath("child", "source", filepath.Join(childDirectory, "child.txt"))
|
||||
// Keep ../ coverage separate from paths that distinguish the declaring file.
|
||||
assertReferencePath("child", "outside", filepath.Join(fixtureDirectory, "outside.txt"))
|
||||
if got := loaded.Config.Pipelines["child"].Steps[0].References["step-source"].Path; got != filepath.Join(parentDirectory, "step.txt") {
|
||||
t.Errorf("inherited step reference path = %q", got)
|
||||
}
|
||||
|
||||
overlaid, err := loadConfig(main, LoadOptions{Overlay: "development"})
|
||||
if err != nil {
|
||||
t.Fatalf("loadConfig() with overlay error = %v", err)
|
||||
}
|
||||
for _, pipelineID := range []string{"parent", "child"} {
|
||||
for _, referenceID := range []string{"shared", "nested"} {
|
||||
if got, want := overlaid.Config.Pipelines[pipelineID].References[referenceID].Path, filepath.Join(overlayDirectory, "overlay.txt"); got != want {
|
||||
t.Errorf("overlaid %s reference %s path = %q, want %q", pipelineID, referenceID, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
if got, want := overlaid.Config.Pipelines["child"].References["source"].Path, filepath.Join(childDirectory, "child.txt"); got != want {
|
||||
t.Errorf("child replacement path = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := overlaid.Config.Pipelines["child"].Steps[0].References["step-source"].Path, filepath.Join(parentDirectory, "step.txt"); got != want {
|
||||
t.Errorf("overlaid inherited step reference path = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInheritedInspectionRetainsOriginsRedactionAndOwnership(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
root := t.TempDir()
|
||||
parentDirectory := filepath.Join(root, "parents")
|
||||
childDirectory := filepath.Join(root, "children")
|
||||
overlayDirectory := filepath.Join(root, "overlays")
|
||||
for _, directory := range []string{parentDirectory, childDirectory, overlayDirectory} {
|
||||
if err := os.MkdirAll(directory, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
parentFile := filepath.Join(parentDirectory, "pipelines.yml")
|
||||
if err := os.WriteFile(parentFile, []byte(`pipelines:
|
||||
parent:
|
||||
input: {module: input}
|
||||
structured_output_repair_attempts: 2
|
||||
references:
|
||||
source: {path: ./shared.txt}
|
||||
artifacts:
|
||||
summary:
|
||||
extract:
|
||||
module: extract
|
||||
options:
|
||||
nested: {value: original}
|
||||
credentials: {api_key: fake-parent-secret}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(parentDirectory, "shared.txt"), []byte("source\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
childFile := filepath.Join(childDirectory, "pipelines.yml")
|
||||
if err := os.WriteFile(childFile, []byte(`pipelines:
|
||||
child:
|
||||
extends: parent
|
||||
structured_output_repair_attempts: 0
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
main := filepath.Join(root, "notarius.yml")
|
||||
if err := os.WriteFile(main, []byte(`version: 5
|
||||
configkit:
|
||||
imports:
|
||||
- path: parents/pipelines.yml
|
||||
- path: children/pipelines.yml
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(overlayDirectory, "development.yml"), []byte(`pipelines:
|
||||
parent:
|
||||
artifacts:
|
||||
summary:
|
||||
extract:
|
||||
options:
|
||||
credentials: {api_key: fake-overlay-secret}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
loaded, err := Load(main, LoadOptions{Overlay: "development"})
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
child := loaded.Config.Pipelines["child"]
|
||||
wantPath := filepath.Join(parentDirectory, "shared.txt")
|
||||
if got := child.References["source"].Path; got != wantPath {
|
||||
t.Fatalf("runtime inherited path = %q, want %q", got, wantPath)
|
||||
}
|
||||
if child.StructuredOutputRepairAttempts == nil || *child.StructuredOutputRepairAttempts != 0 {
|
||||
t.Fatalf("runtime inherited repair setting = %#v", child.StructuredOutputRepairAttempts)
|
||||
}
|
||||
|
||||
var effective bytes.Buffer
|
||||
if err := loaded.WriteEffective(&effective); err != nil {
|
||||
t.Fatalf("WriteEffective() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(effective.String(), wantPath) || strings.Contains(effective.String(), "extends:") ||
|
||||
!strings.Contains(effective.String(), "structured_output_repair_attempts: 0") {
|
||||
t.Fatalf("effective configuration is inconsistent with runtime conversion: %s", effective.String())
|
||||
}
|
||||
for _, secret := range []string{"fake-parent-secret", "fake-overlay-secret"} {
|
||||
if strings.Contains(effective.String(), secret) {
|
||||
t.Fatalf("effective configuration exposed %q", secret)
|
||||
}
|
||||
}
|
||||
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
path configkit.Path
|
||||
}{
|
||||
{name: "repair setting", path: configkit.Keys("pipelines", "child", "structured_output_repair_attempts")},
|
||||
{name: "normalized reference", path: configkit.Keys("pipelines", "child", "references", "source", "path")},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
explanation, ok := loaded.result.Explain(test.path)
|
||||
if !ok || len(explanation.History) == 0 {
|
||||
t.Fatalf("inherited explanation = %#v, found=%v", explanation, ok)
|
||||
}
|
||||
inheritanceFromParent := false
|
||||
for _, change := range explanation.History {
|
||||
if len(change.Origin.Inheritance) == 0 {
|
||||
continue
|
||||
}
|
||||
for _, location := range change.Origin.Locations {
|
||||
for _, source := range loaded.result.Sources() {
|
||||
if source.ID == location.SourceID && source.Path == parentFile {
|
||||
inheritanceFromParent = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !inheritanceFromParent {
|
||||
t.Fatalf("inheritance history lacks parent source evidence: %#v", explanation.History)
|
||||
}
|
||||
var rendered bytes.Buffer
|
||||
if err := loaded.WriteExplanation(&rendered, test.path.String()); err != nil {
|
||||
t.Fatalf("WriteExplanation() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(rendered.String(), filepath.ToSlash(filepath.Join("parents", "pipelines.yml"))) {
|
||||
t.Fatalf("explanation lacks parent source evidence: %s", rendered.String())
|
||||
}
|
||||
if test.name == "normalized reference" {
|
||||
foundNormalization := false
|
||||
for _, change := range explanation.History {
|
||||
if change.Origin.Operation == configkit.ChangeNormalization {
|
||||
foundNormalization = true
|
||||
}
|
||||
}
|
||||
if !foundNormalization || !strings.Contains(rendered.String(), wantPath) {
|
||||
t.Fatalf("normalized path evidence = %#v; rendered %s", explanation.History, rendered.String())
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
secretPath := configkit.Keys("pipelines", "child", "artifacts", "summary", "extract", "options", "credentials", "api_key")
|
||||
secretExplanation, ok := loaded.result.Explain(secretPath)
|
||||
if !ok || !secretExplanation.Effective.Redacted {
|
||||
t.Fatalf("inherited secret explanation = %#v, found=%v", secretExplanation, ok)
|
||||
}
|
||||
redactedHistory := false
|
||||
for _, change := range secretExplanation.History {
|
||||
redactedHistory = redactedHistory || change.Before.Redacted || change.After.Redacted
|
||||
}
|
||||
if !redactedHistory {
|
||||
t.Fatalf("secret history was not redacted: %#v", secretExplanation.History)
|
||||
}
|
||||
var secretOutput bytes.Buffer
|
||||
if err := loaded.WriteExplanation(&secretOutput, secretPath.String()); err != nil {
|
||||
t.Fatalf("WriteExplanation(secret) error = %v", err)
|
||||
}
|
||||
if strings.Contains(secretOutput.String(), "fake-parent-secret") || strings.Contains(secretOutput.String(), "fake-overlay-secret") ||
|
||||
!strings.Contains(secretOutput.String(), "[REDACTED]") {
|
||||
t.Fatalf("secret explanation did not redact values and history: %s", secretOutput.String())
|
||||
}
|
||||
|
||||
childOptions := loaded.Config.Pipelines["child"].Artifacts["summary"].Extract.Options
|
||||
childOptions["nested"].(map[string]any)["value"] = "changed"
|
||||
parentValue := loaded.Config.Pipelines["parent"].Artifacts["summary"].Extract.Options["nested"].(map[string]any)["value"]
|
||||
if parentValue != "original" {
|
||||
t.Fatalf("child runtime mutation changed parent value: %#v", parentValue)
|
||||
}
|
||||
var frozen bytes.Buffer
|
||||
if err := loaded.WriteEffective(&frozen); err != nil {
|
||||
t.Fatalf("WriteEffective() after runtime mutation error = %v", err)
|
||||
}
|
||||
if !strings.Contains(frozen.String(), "original") || strings.Contains(frozen.String(), "changed") {
|
||||
t.Fatalf("runtime mutation changed frozen inspection: %s", frozen.String())
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user