Plan ConfigKit adoption and clarify implementation policy
This commit is contained in:
615
docs/roadmap/configkit.md
Normal file
615
docs/roadmap/configkit.md
Normal file
@@ -0,0 +1,615 @@
|
||||
# ConfigKit Adoption
|
||||
|
||||
## Purpose
|
||||
|
||||
Replace Notarius's application-owned YAML parsing, presence bookkeeping,
|
||||
default application, and configuration-inspection machinery with ConfigKit,
|
||||
while preserving Notarius ownership of pipeline semantics, module catalogs,
|
||||
runtime resolution, and invocation behavior.
|
||||
|
||||
This roadmap is based on ConfigKit v0.2.0, the current stable release in the
|
||||
local `../ConfigKit` checkout. ConfigKit's maintained consumer guide includes a
|
||||
Notarius-shaped example, and its complete test suite passes locally. The
|
||||
dependency starts at v0.2.0; completing source-aware normalization requires a
|
||||
reviewed release with the extension below. Do not follow an unreviewed branch
|
||||
tip. Ordered implementation work belongs in
|
||||
[the implementation plan](implementation.md).
|
||||
|
||||
### Source Metadata Dependency
|
||||
|
||||
Review for implementation identified an API gap in v0.2.0: `Resolution`
|
||||
exposes input presence and change recording, but not declaration origins or
|
||||
source paths. Those are available only on the completed `Result`, whose
|
||||
effective and explanation snapshots are already frozen. The target of
|
||||
recording declaration-relative path normalization in those snapshots therefore
|
||||
requires an upstream extension. The selected approach is to request source
|
||||
origins and canonical source records during resolution, then adopt a reviewed
|
||||
release providing that API. The implementation plan records the dependency
|
||||
gate. Resolving paths only after loading with a separate inspection report was
|
||||
considered but not selected; the agreed effective/explanation contract remains
|
||||
the target. ConfigKit v0.2.0 alone is insufficient for that part of the target.
|
||||
|
||||
## Motivation
|
||||
|
||||
The current configuration boundary under `internal/core/config` works, but it
|
||||
reimplements a substantial general-purpose configuration system. In
|
||||
particular, Notarius currently owns:
|
||||
|
||||
- strict YAML setup, duplicate-key and trailing-document checks;
|
||||
- custom YAML decoders for module bindings, references, pipelines, steps, and
|
||||
lanes;
|
||||
- pointer and private-flag bookkeeping to distinguish omitted values from
|
||||
explicit zero, false, empty, or null values;
|
||||
- a large file-to-runtime application function that combines normalization,
|
||||
validation, conversion, and default derivation;
|
||||
- operational environment overlays and their separate validation path; and
|
||||
- bespoke cloning and redaction for effective configuration summaries.
|
||||
|
||||
ConfigKit already provides strict single-document YAML parsing, typed decode,
|
||||
input presence, typed defaults, recorded normalization and derived defaults,
|
||||
semantic diagnostics with source locations, detached results, redacted
|
||||
effective rendering, and value explanation. Adopting those facilities should
|
||||
reduce application-specific code and make configuration failures easier for an
|
||||
operator to understand without moving domain policy into the dependency.
|
||||
|
||||
This is a configuration-contract migration, not a mechanical parser swap.
|
||||
ConfigKit deliberately rejects several conveniences supported by version 4,
|
||||
and its dynamic integer representation differs from `gopkg.in/yaml.v3`.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- every command uses one ConfigKit-backed loading path;
|
||||
- Notarius configuration files declare schema version 5 using the existing
|
||||
integer YAML convention;
|
||||
- the file schema uses canonical object forms rather than YAML type-dependent
|
||||
shorthand;
|
||||
- operators can import whole pipeline definitions and select a configuration
|
||||
profile for development, testing, or deployment;
|
||||
- defaults, surviving-input presence, normalization, derived values, and
|
||||
configuration-only validation are expressed through one ConfigKit
|
||||
`Definition`;
|
||||
- ConfigKit diagnostics are rendered safely with semantic paths and available
|
||||
source locations at the CLI boundary;
|
||||
- the successful load result retains ConfigKit metadata for safe effective
|
||||
configuration rendering and explanation;
|
||||
- dynamic module-option integers have one documented `int64` configuration
|
||||
representation and are range-checked when converted to runtime `int` values;
|
||||
- pipeline and module resolution remains in Notarius and still fails before
|
||||
module preparation when a configured composition is invalid;
|
||||
- configuration discovery and invocation-only CLI state remain application
|
||||
responsibilities;
|
||||
- `NOTARIUS_CONFIG` remains a configuration-discovery input, while the seven
|
||||
field-level operational environment overlays are retired;
|
||||
- read-only CLI commands expose ConfigKit's redacted effective configuration
|
||||
and per-path provenance explanations; and
|
||||
- the direct `gopkg.in/yaml.v3` dependency and the superseded custom parsing,
|
||||
presence, environment-overlay, and configuration-redaction code are removed.
|
||||
|
||||
The migration must preserve the fixed pipeline architecture, deterministic
|
||||
resolved-pipeline identity, detached ownership, fail-before-execution checks,
|
||||
secret exclusion, source-relative reference behavior, and offline validation
|
||||
requirements defined by the architecture and current integration contracts.
|
||||
|
||||
## Ownership Boundaries
|
||||
|
||||
### ConfigKit owns
|
||||
|
||||
- reading and bounding the selected YAML source, its declared imports, and the
|
||||
selected configuration profile after Notarius chooses them;
|
||||
- composing imported definitions and applying the selected profile overlay;
|
||||
- enforcing a single strict YAML document, supported scalar forms, unique
|
||||
mapping keys, and known typed fields;
|
||||
- decoding into the application-defined configuration schema;
|
||||
- distinguishing omitted input from explicit zero, false, empty, and null;
|
||||
- applying typed application defaults;
|
||||
- recording application normalization and derived defaults;
|
||||
- attaching application validation diagnostics to semantic paths and source
|
||||
locations;
|
||||
- detaching the successful configuration and its metadata;
|
||||
- rendering the redacted effective configuration; and
|
||||
- explaining the provenance of an effective or historical configuration path.
|
||||
|
||||
### Notarius continues to own
|
||||
|
||||
- configuration-file discovery through `--config`, `NOTARIUS_CONFIG`, and the
|
||||
installed default path;
|
||||
- the version policy and the application configuration structs;
|
||||
- the meaning and validation of every Notarius field;
|
||||
- pipeline, module, validator, artifact, and reference catalogs;
|
||||
- conversion from configuration-facing values to runtime pipeline types;
|
||||
- source-relative interpretation of configured external reference paths;
|
||||
- run-time selectors and actions such as lane selection, input, resume,
|
||||
recomputation, session identity, and CLI reference changes;
|
||||
- PromptKit profile discovery and validation; and
|
||||
- redaction of resolved runtime structures that are outside ConfigKit's typed
|
||||
configuration snapshot, including exclusion of materialized reference
|
||||
content.
|
||||
|
||||
ConfigKit configuration profiles must not be confused with PromptKit LLM
|
||||
profiles or Notarius named pipeline definitions. ConfigKit does not become a
|
||||
module registry, pipeline resolver, command framework, or secret manager for
|
||||
PromptKit credentials.
|
||||
|
||||
## Configuration Contract Version 5
|
||||
|
||||
### Version spelling
|
||||
|
||||
Use the following root declaration:
|
||||
|
||||
```yaml
|
||||
version: 5
|
||||
```
|
||||
|
||||
ConfigKit v0.2.0 accepts string or strict decimal-integer input at the exact
|
||||
path selected by its `VersionRule`. It passes the original spelling to the
|
||||
application checker and can decode an accepted integer into a string-backed Go
|
||||
field. This narrow exception lets Notarius retain its existing numeric YAML
|
||||
convention without weakening strict decoding elsewhere. Quoted `version: "5"`
|
||||
is also accepted by ConfigKit, but the maintained Notarius spelling is the
|
||||
unquoted integer shown above.
|
||||
|
||||
The version checker must reject absent, null, and unsupported identifiers with
|
||||
safe, value-free diagnostics at `/version`. Version 5 is a clean break: the
|
||||
loader must not decode or rewrite a version 4 document and must not retain the
|
||||
legacy loader as a fallback.
|
||||
|
||||
### Canonical module bindings
|
||||
|
||||
Every module and validator binding must use an object with a `module` field.
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
input:
|
||||
module: seriatim
|
||||
extract:
|
||||
module: dnd/spells
|
||||
merge:
|
||||
module: appendorder
|
||||
normalize:
|
||||
module: dnd/spells
|
||||
```
|
||||
|
||||
Version 5 does not accept scalar binding shorthand. Preserving shorthand would
|
||||
require a dynamic subtree or a compatibility pre-parser and would retain much
|
||||
of the complexity this adoption is intended to remove.
|
||||
|
||||
### Canonical reference sources
|
||||
|
||||
Reference sources must also use explicit objects. An external file reference
|
||||
uses a `path` field; a generated reference retains its typed `artifact`
|
||||
selector. For example:
|
||||
|
||||
```yaml
|
||||
references:
|
||||
party:
|
||||
path: ./dnd-party.txt
|
||||
npc_registry:
|
||||
artifact:
|
||||
step: describe-session
|
||||
lane: npc-registry
|
||||
```
|
||||
|
||||
Pipeline-level references remain external-file-only. Step- and binding-local
|
||||
references may select earlier artifacts according to the existing ordered
|
||||
handoff rules. Relative external paths resolve from the file that declared the
|
||||
surviving value, including imported fragments and configuration profiles. CLI
|
||||
reference paths continue to resolve from the process working directory.
|
||||
|
||||
Remove the lane-level `references` compatibility alias in version 5. New files
|
||||
must place a lane's extraction references under `extract.references`. This
|
||||
leaves one canonical location and removes merge-precedence behavior from the
|
||||
file converter.
|
||||
|
||||
### Strictness and dynamic options
|
||||
|
||||
Keep ConfigKit's strict YAML rules. Version 5 therefore rejects aliases, merge
|
||||
keys, unsupported tags and scalar forms, duplicate mapping keys, unknown typed
|
||||
fields, trailing documents, and incompatible YAML types. Do not recreate an
|
||||
application pre-parser to weaken these rules.
|
||||
|
||||
Module `options` remain `map[string]any` where the selected module determines
|
||||
the valid keys. ConfigKit represents YAML integers in dynamic values as
|
||||
`int64`. Module option validators must accept that canonical representation,
|
||||
check range before conversion, and hand runtime modules a safe `int` only where
|
||||
their APIs require it. Do not marshal and decode dynamic subtrees through YAML
|
||||
to recover native `int` values.
|
||||
|
||||
## ConfigKit-Backed Loading Model
|
||||
|
||||
Define a plain, typed configuration document for ConfigKit. It should mirror
|
||||
the public file schema, use ordinary YAML tags, and contain no custom YAML
|
||||
unmarshal hooks. Keep it separate from framework runtime types where doing so
|
||||
avoids YAML concerns, presence flags, or configuration-only unions leaking into
|
||||
`internal/framework/pipeline`.
|
||||
|
||||
The reusable ConfigKit definition should provide:
|
||||
|
||||
- fresh built-in defaults for each load;
|
||||
- the `/version` rule;
|
||||
- recorded whitespace normalization for identifiers, paths, profile IDs,
|
||||
endpoints, and other fields whose current contract normalizes input;
|
||||
- recorded derivation of `concurrency.stage_workers.extract` from the effective
|
||||
`concurrency.total_llm` only when the worker value was omitted;
|
||||
- collision checks before normalizing map keys or cross-step identifiers;
|
||||
- configuration-only semantic validation with precise paths and related paths
|
||||
where useful; and
|
||||
- a dynamic secret predicate that covers credential-shaped keys below module
|
||||
option maps, preserving the current defensive redaction boundary even though
|
||||
ordinary Notarius configuration contains no credential values.
|
||||
|
||||
Use ConfigKit's resolver only for deterministic configuration normalization and
|
||||
derived defaults. Use its read-only semantic validator for domain constraints.
|
||||
Do not put module lookup, PromptKit profile lookup, filesystem resource checks,
|
||||
or runtime pipeline construction in either callback.
|
||||
|
||||
After a successful load, perform one explicit, independently tested conversion
|
||||
from the canonical configuration document to the Notarius runtime `Config` and
|
||||
pipeline profile types. The converter should be a narrow ownership boundary,
|
||||
not a second parser or validation framework. It may translate enums and checked
|
||||
numeric values, but it must not silently change already validated meaning.
|
||||
|
||||
Retain the ConfigKit result alongside the runtime configuration in an
|
||||
application-owned loaded-configuration value. Commands that need only the
|
||||
runtime `Config` should not gain access to parser internals; commands and debug
|
||||
writers that expose safe configuration provenance may use the retained result.
|
||||
|
||||
## Imports And Configuration Profiles
|
||||
|
||||
Enable ConfigKit's explicit imports using its existing `configkit.imports`
|
||||
syntax, including path, glob, and optional declarations. Operators can keep
|
||||
whole pipeline definitions in separate files and compose them through the
|
||||
selected root file. The root file alone declares `version`.
|
||||
|
||||
Use ConfigKit's strict composition semantics. Imports combine disjoint
|
||||
definitions; they do not provide ordered override layers or concatenate
|
||||
sequences. In particular, one pipeline's ordered `steps` list remains together
|
||||
in a single definition. Do not add Notarius-specific step merging or a second
|
||||
import mechanism.
|
||||
|
||||
Expose `--config-profile <name>` on every command that loads configuration:
|
||||
`run`, `config validate`, `pipelines list`, `config effective`, and
|
||||
`config explain`. Pass the selection to ConfigKit. A missing selection uses
|
||||
the composed base configuration. ConfigKit's reserved selection `default`
|
||||
also means no overlay; it does not load a default profile file.
|
||||
|
||||
Use ConfigKit's profile location `<configuration-root>/profiles/<name>.yml`
|
||||
and existing selection rules. A selected missing profile fails loading.
|
||||
Profiles recursively overlay mappings and replace scalars, sequences, and
|
||||
nulls as specified upstream. Development profiles can change pipeline-level
|
||||
`llm_profile`, concurrency, and output settings without repeating the pipeline.
|
||||
Changing an individual binding inside `steps` requires replacing the complete
|
||||
steps sequence. Document this limitation prominently in examples.
|
||||
|
||||
Configuration profiles and PromptKit profiles are independent. The former
|
||||
select a configuration overlay with `--config-profile`; the latter select LLM
|
||||
settings through `llm_profile` and the existing `--llm-profile` override. Neither
|
||||
selection implicitly chooses the other.
|
||||
|
||||
Use the selected file's canonical directory as ConfigKit's configuration root.
|
||||
Document the upstream requirement that imports and configuration profiles stay
|
||||
within that root, along with composition conflicts, cycle/repeated-import
|
||||
errors, and profile sequence replacement. This containment rule does not apply
|
||||
to Notarius input, reference, output, cache, debug, or PromptKit profile paths.
|
||||
|
||||
### Paths And Source Ownership
|
||||
|
||||
Resolve relative external reference paths and `promptkit.profile_file` and
|
||||
`promptkit.profile_dir` relative to the file that supplied the surviving value.
|
||||
This deliberately changes PromptKit profile-source paths from their current
|
||||
working-directory convention in version 5. Absolute paths remain absolute.
|
||||
Keep output, cache, and debug relative-directory semantics unchanged; they are
|
||||
runtime destinations rather than files bundled with configuration fragments.
|
||||
|
||||
Use ConfigKit's source metadata to identify each surviving declaration's
|
||||
origin, including profile replacements and values inherited from the base.
|
||||
Record path normalization so effective output and explanation reflect the
|
||||
paths used at runtime. Preserve attribution when identifier normalization
|
||||
renames an enclosing map key. Do not resolve overwritten or removed path
|
||||
values or infer source ownership from diagnostic display labels. Explicit CLI
|
||||
paths retain their working-directory interpretation.
|
||||
|
||||
## Discovery, Precedence, And Overrides
|
||||
|
||||
Keep the existing file-discovery order: explicit `--config`, then non-empty
|
||||
`NOTARIUS_CONFIG`, then the installed default path. `NOTARIUS_CONFIG` selects a
|
||||
source and is not a field-level configuration override, so it remains outside
|
||||
ConfigKit.
|
||||
|
||||
Retire these seven field-level environment overlays in version 5:
|
||||
|
||||
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`;
|
||||
- `NOTARIUS_STAGE_WORKERS_EXTRACT`;
|
||||
- `NOTARIUS_OUTPUT_DIR`;
|
||||
- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`;
|
||||
- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`;
|
||||
- `NOTARIUS_CACHE_CHECKPOINTS_DIR`; and
|
||||
- `NOTARIUS_DEBUG_DIR`.
|
||||
|
||||
ConfigKit deliberately does not model general environment overlays. Applying
|
||||
them after a successful load would make the retained effective configuration,
|
||||
origins, and explanations disagree with the values used at runtime. Operators
|
||||
must put durable values in YAML and use the supported explicit CLI flags for
|
||||
invocation-local changes.
|
||||
|
||||
Classify the current run flags by responsibility. A flag that directly
|
||||
overrides a field in the typed configuration document should become a
|
||||
ConfigKit `Override` record only when the operator explicitly supplied it;
|
||||
parser defaults must not be submitted as overrides. Input selection, lane
|
||||
selection, session identity, resume/recompute actions, command-line references,
|
||||
and similar invocation state remain outside ConfigKit even when they affect
|
||||
runtime resolution. The documented precedence becomes built-in defaults,
|
||||
composed YAML sources, the selected configuration profile, explicit
|
||||
configuration-valued CLI overrides, and then
|
||||
invocation-only resolution actions at their existing application boundary.
|
||||
|
||||
## Validation And Diagnostics
|
||||
|
||||
Move configuration-only checks currently split among parsing, file
|
||||
application, and `Config.Validate` into the ConfigKit
|
||||
definition when they can be expressed against the typed file contract. Report
|
||||
all independently discoverable semantic errors in one validation pass rather
|
||||
than failing at the first field, while avoiding cascaded diagnostics whose only
|
||||
cause is an earlier invalid parent.
|
||||
|
||||
Keep catalog-dependent checks in `Config.Resolve` and
|
||||
`pipeline.ResolvePipeline`. These include registered module and validator
|
||||
existence, stage capabilities, artifact compatibility, reference-slot
|
||||
compatibility, PromptKit profile availability, and the concrete resolved
|
||||
pipeline digest. Their existing error classification and fail-before-prepare
|
||||
behavior must remain intact.
|
||||
|
||||
At the CLI boundary, recognize `*configkit.LoadError` and use ConfigKit's
|
||||
canonical diagnostic renderer. Operators should receive the diagnostic code,
|
||||
semantic path, safe message, and source coordinates when available. Do not
|
||||
append raw YAML parser errors, source fragments, configuration values, or
|
||||
secret values to the outward error chain. File discovery/read context should
|
||||
remain clear without wrapping the same failure repeatedly.
|
||||
|
||||
Nonfatal ConfigKit diagnostics, if Notarius defines any, must not be silently
|
||||
dropped. They should enter the existing process-warning model only when they
|
||||
represent an actionable configuration degradation. Ordinary normalization,
|
||||
defaulting, and provenance records are explanation data, not run warnings.
|
||||
|
||||
## Effective Configuration And Redaction
|
||||
|
||||
Use `Result.WriteEffective` as the canonical renderer for the loaded effective
|
||||
configuration. The requested debug bundle should store this deterministic,
|
||||
redacted YAML rather than a separately maintained JSON clone of the raw
|
||||
configuration. Update the artifact name and documentation together if the
|
||||
format changes.
|
||||
|
||||
Use ConfigKit's secret predicate to redact credential-shaped dynamic option
|
||||
paths. PromptKit credentials remain environment references in PromptKit-owned
|
||||
profile files and must not be copied into Notarius configuration or debug
|
||||
state.
|
||||
|
||||
Do not remove Notarius's redaction of the resolved pipeline merely because the
|
||||
loaded configuration is ConfigKit-backed. The resolved value contains
|
||||
application-owned structures, selection results, and reference provenance that
|
||||
ConfigKit does not own; materialized reference bytes must continue to be
|
||||
excluded from ordinary summaries. Remove only manual configuration cloning and
|
||||
redaction made redundant by the frozen ConfigKit result.
|
||||
|
||||
## Configuration Inspection Commands
|
||||
|
||||
Add these read-only commands as first-class ConfigKit-backed inspection
|
||||
surfaces:
|
||||
|
||||
```text
|
||||
notarius config effective [--config path/to/config.yml] [--config-profile name]
|
||||
notarius config explain [--config path/to/config.yml] [--config-profile name] <json-pointer>
|
||||
```
|
||||
|
||||
`config effective` loads the selected configuration through the production
|
||||
path and writes only ConfigKit's deterministic redacted YAML to standard
|
||||
output. `config explain` accepts one canonical JSON Pointer, loads through the
|
||||
same path, and writes ConfigKit's deterministic redacted explanation for that
|
||||
effective or historical path. Neither command resolves a pipeline, prepares a
|
||||
module, requires provider credentials, or contacts a provider.
|
||||
|
||||
Both commands must use ordinary configuration discovery, ConfigKit diagnostic
|
||||
rendering, the same import and profile selection as execution, and the existing
|
||||
CLI exit-status conventions. An unknown or
|
||||
noncanonical explanation path is a command failure, not an empty successful
|
||||
result. Standard output contains only the requested inspection output;
|
||||
diagnostics go to standard error. Effective output is YAML and explanation is
|
||||
ConfigKit's human-readable text; neither is promised to be reloadable
|
||||
configuration or a new durable JSON API. There is no unredacted mode.
|
||||
|
||||
## Required Code And Contract Changes
|
||||
|
||||
The implementation must address the following coherent work areas:
|
||||
|
||||
- add and pin a reviewed `gitea.maximumdirect.net/eric/ConfigKit` release with
|
||||
v0.2.0's semantics and the required source-origin API;
|
||||
- introduce the version 5 typed document and its ConfigKit definition;
|
||||
- replace `LoadFileConfig`, custom `UnmarshalYAML` implementations, manual
|
||||
strict-parser checks, presence flags, `ApplyFileConfig`, and redundant
|
||||
configuration cloning with the ConfigKit load boundary;
|
||||
- consolidate defaults, normalization, and configuration-only validation into
|
||||
the definition while retaining runtime catalog validation;
|
||||
- update CLI configuration loading and error rendering without changing file
|
||||
discovery;
|
||||
- enable ConfigKit imports and configuration profiles, wire `--config-profile`
|
||||
through all configuration-loading commands, and retain declaration origins
|
||||
for relative file paths;
|
||||
- retire field-level environment overlays while retaining `NOTARIUS_CONFIG`;
|
||||
- classify current run flags so only true configuration overrides use
|
||||
ConfigKit override records and invocation-only state remains outside the
|
||||
configuration loader;
|
||||
- update dynamic module-option decoding and range checks for `int64`;
|
||||
- integrate ConfigKit effective rendering and metadata into requested debug
|
||||
bundles and the new `config effective` and `config explain` commands;
|
||||
- remove superseded code, tests, and the direct `gopkg.in/yaml.v3` dependency
|
||||
after no production or test-only use remains; and
|
||||
- update maintained examples and canonical documentation in the same change as
|
||||
each externally visible contract adjustment.
|
||||
|
||||
## Documentation And Migration
|
||||
|
||||
Update `docs/config.md` as the canonical owner of version 5 syntax, defaults,
|
||||
precedence, canonical bindings and references, strict YAML behavior, and the
|
||||
retirement of field-level environment overrides. Replace the version 3
|
||||
migration section with a concise version 4-to-5 migration guide covering:
|
||||
|
||||
- changing the numeric version value from 4 to 5;
|
||||
- expanding scalar module bindings into `{module: ...}` objects;
|
||||
- expanding scalar external references into `{path: ...}` objects;
|
||||
- moving lane-level reference aliases to `extract.references`;
|
||||
- resolving configured PromptKit profile-source paths from their declaring
|
||||
file instead of the working directory;
|
||||
- using imports for whole pipeline definitions and configuration profiles for
|
||||
overlays, including complete replacement of step sequences;
|
||||
- replacing retired operational environment overrides with YAML or supported
|
||||
explicit CLI flags; and
|
||||
- any effective/debug artifact or inspection-command changes.
|
||||
|
||||
Update `docs/internal/configuration.md` to describe the ConfigKit boundary,
|
||||
typed document-to-runtime conversion, application-owned validation, retained
|
||||
metadata, and catalog-resolution handoff. Update `docs/cli.md` only for changed
|
||||
commands or diagnostic behavior, `docs/operations.md` only for operator-facing
|
||||
configuration and debug workflows, and the relevant architecture text if the
|
||||
adopted precedence model changes.
|
||||
|
||||
Convert both maintained D&D configurations and every configuration snippet to
|
||||
version 5 canonical syntax. Examples must remain secret-free and load through
|
||||
the same production path exercised by the CLI. Do not keep a second set of
|
||||
version 4 examples.
|
||||
|
||||
Keep the minimal D&D example self-contained. Extend the complete D&D example
|
||||
with an imported whole pipeline definition and a selectable development
|
||||
configuration profile. Keep configuration overlays physically distinct from
|
||||
the maintained PromptKit profile files so readers can see which system owns
|
||||
each file. Test the complete example both with and without the development
|
||||
overlay, and update subprocess guidance to show optional `--config-profile`
|
||||
selection. All configuration inspection commands must document what the
|
||||
selected profile changes and that runtime pipeline resolution is separate.
|
||||
|
||||
Because this change establishes a durable dependency and reallocates ownership
|
||||
at a core boundary, add an ADR recording the decision to use ConfigKit, the
|
||||
responsibilities retained by Notarius, the deliberate version 5 break, and the
|
||||
retirement of the application environment-overlay layer. The rejected
|
||||
alternatives should include continuing the custom loader, retaining version 4
|
||||
temporarily, or wrapping ConfigKit in a compatibility pre-parser. The ADR
|
||||
should record rationale, not duplicate the current configuration reference or
|
||||
implementation details.
|
||||
|
||||
## Testing And Verification
|
||||
|
||||
Follow the repository's behavior-focused testing policy. Reuse ConfigKit's own
|
||||
tests for generic YAML and metadata mechanics; Notarius tests should protect
|
||||
only its contract and integration responsibilities.
|
||||
|
||||
Required Notarius coverage includes:
|
||||
|
||||
- successful loading of omitted, explicit zero/false/empty, and derived values
|
||||
that materially affect Notarius behavior;
|
||||
- numeric and quoted version 5 acceptance and safe rejection of absent, null,
|
||||
version 4, unsupported identifiers, and non-integer numeric forms;
|
||||
- canonical binding and reference decoding plus rejection of former shorthand;
|
||||
- identifier normalization and normalized-collision rejection;
|
||||
- aggregation and source location of representative application validation
|
||||
errors;
|
||||
- unchanged configuration discovery and source-relative reference resolution;
|
||||
- imported whole pipeline definitions, representative composition conflicts,
|
||||
and profile precedence through the real ConfigKit loader;
|
||||
- profile sequence replacement and explicit clearing, so Notarius does not
|
||||
accidentally append or merge ordered steps;
|
||||
- relative reference and PromptKit profile-source paths from root files,
|
||||
imported fragments, and overlays, including an unchanged inherited value,
|
||||
a replaced value, and an enclosing normalized identifier;
|
||||
- matching configuration-profile selection across execution, validation,
|
||||
pipeline listing, effective output, and explanation, without provider calls;
|
||||
- checked dynamic integer conversion at each production module option boundary;
|
||||
- detached ownership of loaded configuration and resolved pipelines;
|
||||
- safe CLI rendering of representative ConfigKit parse, decode, version,
|
||||
resolution, and validation failures;
|
||||
- successful redacted `config effective` output and effective, historical,
|
||||
unknown, and noncanonical `config explain` path behavior;
|
||||
- effective-configuration redaction, including nested credential-shaped dynamic
|
||||
option keys, without exposing materialized references;
|
||||
- maintained minimal and complete example validation through the production
|
||||
loader and production module catalog; and
|
||||
- representative end-to-end CLI validation and pipeline resolution without
|
||||
network access or credentials.
|
||||
|
||||
Delete or rewrite legacy parser tests whose only purpose is to retest behavior
|
||||
owned by ConfigKit. Retain a small integration sample for each strictness rule
|
||||
that is itself part of the Notarius public contract, and avoid assertions on
|
||||
incidental diagnostic prose.
|
||||
|
||||
The completed implementation must pass:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
It must also validate both maintained example files with the production CLI.
|
||||
The complete example may use a placeholder credential for offline PromptKit
|
||||
profile validation, as documented today; no live provider call is required.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Moving pipeline resolution, module catalogs, or PromptKit profiles into
|
||||
ConfigKit.
|
||||
- Adding arbitrary environment-to-field binding.
|
||||
- Retaining the retired field-level operational environment variables as a
|
||||
parallel precedence layer.
|
||||
- Loading credentials into the Notarius configuration document.
|
||||
- Supporting remote configuration sources or formats other than YAML.
|
||||
- Adding a generic dynamic-subtree conversion layer.
|
||||
- Composing ordered steps across fragments or adding application-specific
|
||||
sequence merge, profile inheritance, or profile stacking semantics.
|
||||
- Preserving scalar shorthand with an application pre-parser.
|
||||
- Changing the fixed pipeline shape or durable extraction artifacts.
|
||||
- Changing run input, output publication, cache, checkpoint, retry, or warning
|
||||
semantics except where configuration loading or debug rendering directly
|
||||
exposes them.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- All configuration-loading commands use a pinned ConfigKit release providing
|
||||
the required source-origin API, building on v0.2.0, and no production code
|
||||
directly imports a YAML parser.
|
||||
- Version 5 is the only accepted file contract, documents `version: 5` as its
|
||||
canonical spelling, and has one canonical object representation for module
|
||||
bindings and references.
|
||||
- The ConfigKit definition is the sole owner of typed defaults, input presence,
|
||||
recorded normalization, derived configuration defaults, and
|
||||
configuration-only semantic diagnostics.
|
||||
- Notarius remains the sole owner of catalog-dependent resolution and runtime
|
||||
conversion.
|
||||
- Dynamic option integers are handled without unchecked narrowing or YAML
|
||||
round-trips.
|
||||
- CLI diagnostics and effective configuration output are deterministic,
|
||||
redacted, value-safe, and retain useful path/source context.
|
||||
- `config effective` and `config explain` expose only ConfigKit's redacted
|
||||
inspection views and perform no pipeline or provider work.
|
||||
- Imports compose whole pipeline definitions and selected configuration
|
||||
profiles apply ConfigKit's documented overlay semantics consistently across
|
||||
all configuration-loading commands.
|
||||
- Relative external references and PromptKit profile-source paths use the
|
||||
surviving declaration's source directory, and safe inspection agrees with
|
||||
the paths used at runtime.
|
||||
- The resolved pipeline remains detached, reference content remains excluded
|
||||
from ordinary summaries, and its identity changes under the same semantic
|
||||
conditions as before.
|
||||
- Maintained examples, current documentation, focused tests, and repository-wide
|
||||
validation agree with the implemented version 5 contract.
|
||||
- `NOTARIUS_CONFIG` still participates in discovery, while the seven
|
||||
field-level operational environment variables no longer affect loaded
|
||||
configuration.
|
||||
- Superseded custom parser, presence, environment, clone, redaction, and test
|
||||
code is removed rather than left as an alternate loading path.
|
||||
|
||||
## Deferred Composition Work
|
||||
|
||||
Step-level composition may be revisited after operators use imported whole
|
||||
pipeline definitions. It is not part of this adoption: ConfigKit does not
|
||||
concatenate sequences or merge their elements by application-defined IDs.
|
||||
Any future extension must define ordering, duplicate step IDs, dependency
|
||||
validation, profile overrides, and provenance before changing that contract.
|
||||
@@ -1,236 +0,0 @@
|
||||
# Deployment-Managed Cache Permissions
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Summary
|
||||
|
||||
Stop treating cache access permissions as application policy. Notarius should
|
||||
create chunk-plan and checkpoint state through conventional Unix filesystem
|
||||
semantics and leave the effective owner, group, and access policy to the
|
||||
operator's provisioned directories, process umask, setgid inheritance, and
|
||||
default ACLs.
|
||||
|
||||
Notarius continues to own safe paths, symlink rejection, atomic publication,
|
||||
cache validation, and recovery. It does not decide whether a configured cache
|
||||
is private to one account, shared by a trusted group, or readable under some
|
||||
other deployment policy.
|
||||
|
||||
## Motivation And Current Problem
|
||||
|
||||
The current cache implementations enforce a policy that was stronger than the
|
||||
application requires:
|
||||
|
||||
- the chunk-plan store creates and repeatedly converges cache directories to
|
||||
`0700` and atomic files to `0600`; and
|
||||
- the checkpoint recorder supplies `0700` directory and `0600` file modes to
|
||||
the shared confined writer, which explicitly applies the file mode during
|
||||
atomic publication.
|
||||
|
||||
This prevents an administrator from using ordinary Unix provisioning to share
|
||||
one cache between a service account and trusted operator accounts. Adding a
|
||||
second Notarius-specific `private` or `group` configuration policy would solve
|
||||
that immediate symptom, but it would preserve the underlying ownership error:
|
||||
Notarius would still decide access policy instead of respecting the deployment.
|
||||
|
||||
Routine Notarius operation does not require an application-enforced private
|
||||
cache. Cache records can contain source-derived data, so operators must be told
|
||||
what they are storing, but sensitivity does not imply that Notarius should
|
||||
override a deliberately configured filesystem policy. API credentials remain
|
||||
outside cache payloads and continue to follow their existing configuration and
|
||||
redaction boundaries.
|
||||
|
||||
## Target Policy
|
||||
|
||||
The target contract is:
|
||||
|
||||
> Notarius creates cache entries with conventional maximum creation modes and
|
||||
> allows the operating system to determine effective access through the
|
||||
> process umask, inherited group ownership, and default ACLs.
|
||||
|
||||
Concretely:
|
||||
|
||||
- newly created cache directories request `0777`;
|
||||
- newly created cache files, including atomic temporary files, request `0666`;
|
||||
- the kernel applies the invoking process's umask and any applicable default
|
||||
ACL;
|
||||
- setgid directories provisioned by the operator control inherited group
|
||||
ownership and, on supported Unix systems, setgid propagation to descendants;
|
||||
- Notarius does not call `chmod`, `chown`, or `chgrp` to impose or repair a
|
||||
cache access policy; and
|
||||
- Notarius does not reject a cache merely because its effective modes are more
|
||||
or less permissive than a Notarius-selected default.
|
||||
|
||||
This is policy inheritance, not literal copying of a parent directory's mode
|
||||
bits. Directories and non-executable data files appropriately have different
|
||||
maximum modes. A parent mode alone also does not determine child modes: an
|
||||
operator who needs a particular result must configure the service umask and,
|
||||
where appropriate, a setgid parent or default ACL.
|
||||
|
||||
For example, a service started with umask `0077` ordinarily creates `0700`
|
||||
directories and `0600` files. A trusted-group deployment can provision a
|
||||
setgid cache root and use umask `0007`, or an equivalent default ACL, to obtain
|
||||
`0770` directories and `0660` files. These are deployment choices rather than
|
||||
configuration values interpreted by Notarius.
|
||||
|
||||
## Public Contract
|
||||
|
||||
No new Notarius configuration field, environment variable, or CLI flag is
|
||||
introduced. In particular, do not add the previously considered
|
||||
`cache.access: private|group` setting.
|
||||
|
||||
Existing cache enablement, locations, chunk-plan modes, resume controls,
|
||||
identity, and compatibility rules remain unchanged. The behavioral change is
|
||||
that effective permissions on newly created cache entries are controlled by
|
||||
the deployment environment rather than forcibly converged by Notarius.
|
||||
|
||||
The policy applies uniformly to both cache families:
|
||||
|
||||
- canonical chunk plans; and
|
||||
- invocation checkpoints.
|
||||
|
||||
Output and debug storage are outside this feature's scope. Their current
|
||||
contracts must not change incidentally through a shared helper refactor. A
|
||||
separate review may later consider whether the same deployment-owned policy is
|
||||
appropriate for those surfaces.
|
||||
|
||||
## Filesystem And Atomic-Publication Requirements
|
||||
|
||||
Permission deference must cover every cache write path, not only final files:
|
||||
|
||||
- configured cache-family roots created by Notarius;
|
||||
- chunk-plan source-digest directories;
|
||||
- checkpoint identity and stage directories;
|
||||
- atomic temporary files; and
|
||||
- final files published by rename.
|
||||
|
||||
Atomic publication must not bypass the deployment policy. Go's
|
||||
`os.CreateTemp` creates a file with `0600`, and applying `chmod(0666)` afterward
|
||||
would bypass the process umask. Cache writers therefore need an atomic
|
||||
temporary-file creation path that chooses a collision-resistant name and opens
|
||||
it with `O_CREATE|O_EXCL` and requested mode `0666`, allowing the kernel to
|
||||
apply umask and default ACL policy at creation time.
|
||||
|
||||
Replacement writes publish a newly created temporary inode. Its effective
|
||||
permissions and ACL are therefore those selected for new entries by the
|
||||
current parent and process policy; Notarius does not copy mode bits or ACLs
|
||||
from the replaced file. Operators who require stable replacement permissions
|
||||
must express that policy through a stable service umask and directory default
|
||||
ACL.
|
||||
|
||||
Opening an existing cache for writing must not alter its root or descendant
|
||||
modes. Read-only lookup of a missing cache must remain side-effect free.
|
||||
|
||||
## Preserved Safety And Correctness Boundaries
|
||||
|
||||
Deferring access policy does not weaken Notarius's filesystem correctness
|
||||
rules. The implementation must preserve:
|
||||
|
||||
- confined resolution beneath the selected cache root;
|
||||
- rejection of symlinked or unexpected cache entries;
|
||||
- validation of source digests, checkpoint identities, schemas, and recorded
|
||||
provenance;
|
||||
- atomic replacement, existing synchronization behavior, cleanup of abandoned
|
||||
temporary files, and complete-record visibility;
|
||||
- existing missing, malformed, incompatible, and operational-error recovery
|
||||
classifications; and
|
||||
- separation of output, cache, and debug roots.
|
||||
|
||||
An administrator's decision to permit multiple accounts to reach one cache
|
||||
does not create a new Notarius guarantee for simultaneous writers. Preserve
|
||||
the existing atomicity and validation behavior, but do not add locking or claim
|
||||
transactional coordination across concurrent processes as part of this work.
|
||||
Concurrent writes to the same multi-file checkpoint identity remain subject to
|
||||
the currently documented cache behavior; the implementation documentation
|
||||
must state any existing limitation accurately.
|
||||
|
||||
## Architectural Boundaries
|
||||
|
||||
The target removes application-owned cache modes without creating a general
|
||||
permission-policy framework. Chunk-plan and checkpoint storage may use
|
||||
different filesystem APIs internally, but both must implement the same
|
||||
deployment-owned result. Pipeline modules and the runner remain unaware of
|
||||
physical roots and filesystem access policy.
|
||||
|
||||
Any shared writer API must distinguish an application-enforced exact mode from
|
||||
an umask-filtered maximum creation mode. The cache change must not silently
|
||||
alter output or debug behavior merely because checkpoints currently share
|
||||
filesystem helpers with those surfaces.
|
||||
|
||||
## Existing Cache Migration
|
||||
|
||||
Stopping permission convergence does not change existing inode modes. A cache
|
||||
tree created by an older Notarius version may therefore remain `0700`/`0600`
|
||||
and inaccessible to another account after upgrade.
|
||||
|
||||
Operators who want a different policy must either:
|
||||
|
||||
- remove the reconstructible cache and allow it to be recreated beneath a
|
||||
correctly provisioned parent and service environment; or
|
||||
- stop writers and deliberately update ownership, modes, and ACLs using normal
|
||||
system-administration tools.
|
||||
|
||||
Notarius must not perform a recursive startup migration. The same rule applies
|
||||
when an operator wants to make a previously shared cache private: change the
|
||||
deployment policy and deliberately migrate or recreate existing state when an
|
||||
immediate whole-tree change is required.
|
||||
|
||||
## Documentation And Decision Records
|
||||
|
||||
The final documentation set includes an accepted ADR titled along the lines of
|
||||
"Defer cache access policy to the deployment." The ADR records the durable
|
||||
ownership boundary so later maintenance does not reintroduce
|
||||
application-enforced modes solely because cache data can contain source-derived
|
||||
content.
|
||||
|
||||
The new ADR need not supersede ADR-0005 or ADR-0007: those decisions govern
|
||||
canonical chunk-plan reuse and checkpoint recording/reuse, while leaving exact
|
||||
storage permissions to operational contracts. It should reference those ADRs
|
||||
and the output/cache/debug separation established by ADR-0006 and refined by
|
||||
ADR-0007.
|
||||
|
||||
In the final current-behavior documentation, `docs/operations.md` owns cache
|
||||
provisioning, umask, setgid, default ACL, migration, sensitivity, and
|
||||
concurrent-use guidance. Architecture owns the deployment/application boundary;
|
||||
the configuration reference clarifies that selecting a cache path does not
|
||||
select its permissions; and the internal state guide explains the implemented
|
||||
write mechanics. The release that first contains the feature summarizes the
|
||||
operational change without becoming its canonical reference.
|
||||
|
||||
Current-behavior documentation must not describe this target behavior before
|
||||
the implementation lands.
|
||||
|
||||
## Verification Expectations
|
||||
|
||||
Behavioral tests protect kernel-selected permissions for both cache families,
|
||||
non-mutation of existing roots, atomic replacement, and representative
|
||||
assembled CLI wiring. Existing confinement, symlink, validation, identity,
|
||||
reuse, and recovery coverage remains authoritative. Umask tests are POSIX-only,
|
||||
serialized, and restore process-global state. The default suite does not create
|
||||
privileged accounts or groups; setgid and default-ACL examples may be verified
|
||||
in a purpose-built Linux deployment environment.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- No cache access-mode field or application-specific private/group policy is
|
||||
added.
|
||||
- Both chunk-plan and checkpoint writers request `0777` for new directories
|
||||
and `0666` for new files, including atomic temporary files.
|
||||
- Effective permissions reflect the process umask and applicable default ACLs
|
||||
rather than a later Notarius chmod.
|
||||
- Existing cache roots and descendants are not chmod-converged during reads or
|
||||
writes.
|
||||
- A private deployment can obtain private cache state through its filesystem
|
||||
provisioning and umask without a Notarius-specific setting.
|
||||
- A trusted-group deployment can obtain shared cache state through setgid,
|
||||
umask, and/or default ACL policy without a Notarius-specific setting.
|
||||
- Existing cache trees are not recursively migrated.
|
||||
- Chunk-plan and checkpoint enablement, lookup, recording, resume, identity,
|
||||
validation, confinement, atomicity, and recovery semantics remain unchanged.
|
||||
- Output and debug permission behavior does not change as an incidental effect.
|
||||
- Notarius performs no user/group management, ownership mutation, ACL
|
||||
management, privilege escalation, or cross-process locking.
|
||||
- The ADR and canonical operations, architecture, configuration, and internal
|
||||
documentation accurately describe the implemented boundary; the next
|
||||
release note summarizes the change when that release is prepared.
|
||||
- Focused tests and repository-wide `go test ./...`, `go vet ./...`, and
|
||||
`go build ./cmd/notarius` checks pass.
|
||||
@@ -150,16 +150,6 @@ address Go dependency cycles, registrar ownership, stable public module keys,
|
||||
and supersession of the affected ADR-0004 decision. Conceptual artifact-family
|
||||
ownership does not by itself require this move.
|
||||
|
||||
## Near-Term Operations
|
||||
|
||||
### Deployment-Managed Cache Permissions
|
||||
|
||||
- Implement the proposed
|
||||
[deployment-managed cache permissions](deployment-managed-cache-permissions.md)
|
||||
contract so Notarius respects the access policy established by the cache
|
||||
root, process umask, setgid inheritance, and default ACLs instead of imposing
|
||||
application-owned private or group modes.
|
||||
|
||||
## Blue-Sky Platform And Operations
|
||||
|
||||
These ideas are intentionally less specified. Promote one into an earlier
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user