From 00972e8d4835f605319a53bd3687eacc7942f59c Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 12 Sep 2026 02:39:03 +0000 Subject: [PATCH] Plan ConfigKit adoption and clarify implementation policy --- AGENTS.md | 5 + docs/policy/architecture.md | 30 + docs/policy/documentation.md | 12 + docs/policy/testing.md | 7 + docs/roadmap/configkit.md | 615 ++++++++++ .../deployment-managed-cache-permissions.md | 236 ---- docs/roadmap/future.md | 10 - docs/roadmap/implementation.md | 1003 +++++++++++------ 8 files changed, 1342 insertions(+), 576 deletions(-) create mode 100644 docs/roadmap/configkit.md delete mode 100644 docs/roadmap/deployment-managed-cache-permissions.md diff --git a/AGENTS.md b/AGENTS.md index 34531ca6..355653ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,2 +1,7 @@ Please review `docs/development.md` for initial orientation in this repository and follow its task-specific reading guide. + +For every planning and implementation task, follow the +[`Implementation Discipline`](docs/policy/architecture.md#implementation-discipline): +keep machinery proportionate, do not invent restrictions, and do not elevate +incidental implementation details into repository policy. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 290a72c9..9ab8c7ee 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -38,6 +38,36 @@ Notarius is contract-first without being abstraction-heavy. Interfaces and extension points should protect demonstrated boundaries. New abstraction is not itself an architectural goal. +## Implementation Discipline + +Simplify the mechanism, not the user's choices. + +**Keep implementation machinery proportionate.** Use straightforward, +idiomatic code and introduce only the machinery needed to support the requested +behavior and flexibility. Optimize for conceptual simplicity and +maintainability, not for the fewest lines, fewest options, or narrowest product +surface. + +**Do not invent restrictions.** Do not reject, prohibit, constrain, cap, +harden, or otherwise narrow behavior unless required by the user, an explicit +repository contract, or an unavoidable correctness constraint. Preserve +operator and caller choice where policy has not been established. If an +additional restriction appears advisable, present it as a decision rather than +silently implementing it. + +**An implementation detail is not repository policy.** Tests and documentation +must not elevate incidental behavior into a durable guarantee or restriction +unless that behavior was expressly selected as part of the contract. Existing +explicit contracts remain authoritative; changing or weakening one requires a +deliberate decision rather than being inferred from this principle. + +[ADR-0017](../adr/0017-defer-cache-access-policy-to-deployment.md) illustrates +these principles. Notarius previously elevated private cache creation modes +into application policy even though Unix deployments already express access +policy through directory permissions, process umask, setgid inheritance, and +default ACLs. The revised design retains application-owned correctness +boundaries while leaving deployment security policy to the operator. + ## Layers And Dependency Direction The application boundary is the composition root and may depend on concrete diff --git a/docs/policy/documentation.md b/docs/policy/documentation.md index 74fff09a..2b196ea1 100644 --- a/docs/policy/documentation.md +++ b/docs/policy/documentation.md @@ -30,6 +30,18 @@ that the behavior exists. The roadmap owns implementation status and sequencing until the decision is implemented. Current architecture, user, operator, integration, and internal documentation are updated when the behavior lands. +### Implementation Details And Policy + +Documentation must distinguish an explicit contract from an incidental +implementation choice. Do not describe a restriction, guarantee, security +policy, default, or limit as authoritative merely because the current code +implements it. Documentation may accurately describe current behavior without +making it a permanent requirement; distinguish descriptive behavior from +normative policy. The +[implementation-discipline rules](architecture.md#implementation-discipline) +own the criteria for adopting product restrictions and for elevating behavior +into repository policy. + ### Audience And Detail Write for the document's stated audience and include only the detail needed for diff --git a/docs/policy/testing.md b/docs/policy/testing.md index 2c33178e..10097bbe 100644 --- a/docs/policy/testing.md +++ b/docs/policy/testing.md @@ -87,6 +87,13 @@ Do not force all behavior through oversized end-to-end tests. Do not test every A test should protect a decision, contract, or invariant—not memorialize the current implementation. +Tests must not create a stronger product contract than the requirement they +protect. In particular, do not turn an incidental limit, restriction, +permission choice, validation rule, or implementation default into policy by +asserting it without an expressly selected contract. Follow the +[implementation-discipline rules](architecture.md#implementation-discipline) +when deciding what behavior deserves durable protection. + Before adding or retaining a test, ask: > What realistic defect would this test catch? diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md new file mode 100644 index 00000000..d8ed11c2 --- /dev/null +++ b/docs/roadmap/configkit.md @@ -0,0 +1,615 @@ +# ConfigKit Adoption + +## Purpose + +Replace Notarius's application-owned YAML parsing, presence bookkeeping, +default application, and configuration-inspection machinery with ConfigKit, +while preserving Notarius ownership of pipeline semantics, module catalogs, +runtime resolution, and invocation behavior. + +This roadmap is based on ConfigKit v0.2.0, the current stable release in the +local `../ConfigKit` checkout. ConfigKit's maintained consumer guide includes a +Notarius-shaped example, and its complete test suite passes locally. The +dependency starts at v0.2.0; completing source-aware normalization requires a +reviewed release with the extension below. Do not follow an unreviewed branch +tip. Ordered implementation work belongs in +[the implementation plan](implementation.md). + +### Source Metadata Dependency + +Review for implementation identified an API gap in v0.2.0: `Resolution` +exposes input presence and change recording, but not declaration origins or +source paths. Those are available only on the completed `Result`, whose +effective and explanation snapshots are already frozen. The target of +recording declaration-relative path normalization in those snapshots therefore +requires an upstream extension. The selected approach is to request source +origins and canonical source records during resolution, then adopt a reviewed +release providing that API. The implementation plan records the dependency +gate. Resolving paths only after loading with a separate inspection report was +considered but not selected; the agreed effective/explanation contract remains +the target. ConfigKit v0.2.0 alone is insufficient for that part of the target. + +## Motivation + +The current configuration boundary under `internal/core/config` works, but it +reimplements a substantial general-purpose configuration system. In +particular, Notarius currently owns: + +- strict YAML setup, duplicate-key and trailing-document checks; +- custom YAML decoders for module bindings, references, pipelines, steps, and + lanes; +- pointer and private-flag bookkeeping to distinguish omitted values from + explicit zero, false, empty, or null values; +- a large file-to-runtime application function that combines normalization, + validation, conversion, and default derivation; +- operational environment overlays and their separate validation path; and +- bespoke cloning and redaction for effective configuration summaries. + +ConfigKit already provides strict single-document YAML parsing, typed decode, +input presence, typed defaults, recorded normalization and derived defaults, +semantic diagnostics with source locations, detached results, redacted +effective rendering, and value explanation. Adopting those facilities should +reduce application-specific code and make configuration failures easier for an +operator to understand without moving domain policy into the dependency. + +This is a configuration-contract migration, not a mechanical parser swap. +ConfigKit deliberately rejects several conveniences supported by version 4, +and its dynamic integer representation differs from `gopkg.in/yaml.v3`. + +## Target End State + +After this work: + +- every command uses one ConfigKit-backed loading path; +- Notarius configuration files declare schema version 5 using the existing + integer YAML convention; +- the file schema uses canonical object forms rather than YAML type-dependent + shorthand; +- operators can import whole pipeline definitions and select a configuration + profile for development, testing, or deployment; +- defaults, surviving-input presence, normalization, derived values, and + configuration-only validation are expressed through one ConfigKit + `Definition`; +- ConfigKit diagnostics are rendered safely with semantic paths and available + source locations at the CLI boundary; +- the successful load result retains ConfigKit metadata for safe effective + configuration rendering and explanation; +- dynamic module-option integers have one documented `int64` configuration + representation and are range-checked when converted to runtime `int` values; +- pipeline and module resolution remains in Notarius and still fails before + module preparation when a configured composition is invalid; +- configuration discovery and invocation-only CLI state remain application + responsibilities; +- `NOTARIUS_CONFIG` remains a configuration-discovery input, while the seven + field-level operational environment overlays are retired; +- read-only CLI commands expose ConfigKit's redacted effective configuration + and per-path provenance explanations; and +- the direct `gopkg.in/yaml.v3` dependency and the superseded custom parsing, + presence, environment-overlay, and configuration-redaction code are removed. + +The migration must preserve the fixed pipeline architecture, deterministic +resolved-pipeline identity, detached ownership, fail-before-execution checks, +secret exclusion, source-relative reference behavior, and offline validation +requirements defined by the architecture and current integration contracts. + +## Ownership Boundaries + +### ConfigKit owns + +- reading and bounding the selected YAML source, its declared imports, and the + selected configuration profile after Notarius chooses them; +- composing imported definitions and applying the selected profile overlay; +- enforcing a single strict YAML document, supported scalar forms, unique + mapping keys, and known typed fields; +- decoding into the application-defined configuration schema; +- distinguishing omitted input from explicit zero, false, empty, and null; +- applying typed application defaults; +- recording application normalization and derived defaults; +- attaching application validation diagnostics to semantic paths and source + locations; +- detaching the successful configuration and its metadata; +- rendering the redacted effective configuration; and +- explaining the provenance of an effective or historical configuration path. + +### Notarius continues to own + +- configuration-file discovery through `--config`, `NOTARIUS_CONFIG`, and the + installed default path; +- the version policy and the application configuration structs; +- the meaning and validation of every Notarius field; +- pipeline, module, validator, artifact, and reference catalogs; +- conversion from configuration-facing values to runtime pipeline types; +- source-relative interpretation of configured external reference paths; +- run-time selectors and actions such as lane selection, input, resume, + recomputation, session identity, and CLI reference changes; +- PromptKit profile discovery and validation; and +- redaction of resolved runtime structures that are outside ConfigKit's typed + configuration snapshot, including exclusion of materialized reference + content. + +ConfigKit configuration profiles must not be confused with PromptKit LLM +profiles or Notarius named pipeline definitions. ConfigKit does not become a +module registry, pipeline resolver, command framework, or secret manager for +PromptKit credentials. + +## Configuration Contract Version 5 + +### Version spelling + +Use the following root declaration: + +```yaml +version: 5 +``` + +ConfigKit v0.2.0 accepts string or strict decimal-integer input at the exact +path selected by its `VersionRule`. It passes the original spelling to the +application checker and can decode an accepted integer into a string-backed Go +field. This narrow exception lets Notarius retain its existing numeric YAML +convention without weakening strict decoding elsewhere. Quoted `version: "5"` +is also accepted by ConfigKit, but the maintained Notarius spelling is the +unquoted integer shown above. + +The version checker must reject absent, null, and unsupported identifiers with +safe, value-free diagnostics at `/version`. Version 5 is a clean break: the +loader must not decode or rewrite a version 4 document and must not retain the +legacy loader as a fallback. + +### Canonical module bindings + +Every module and validator binding must use an object with a `module` field. +For example: + +```yaml +input: + module: seriatim +extract: + module: dnd/spells +merge: + module: appendorder +normalize: + module: dnd/spells +``` + +Version 5 does not accept scalar binding shorthand. Preserving shorthand would +require a dynamic subtree or a compatibility pre-parser and would retain much +of the complexity this adoption is intended to remove. + +### Canonical reference sources + +Reference sources must also use explicit objects. An external file reference +uses a `path` field; a generated reference retains its typed `artifact` +selector. For example: + +```yaml +references: + party: + path: ./dnd-party.txt + npc_registry: + artifact: + step: describe-session + lane: npc-registry +``` + +Pipeline-level references remain external-file-only. Step- and binding-local +references may select earlier artifacts according to the existing ordered +handoff rules. Relative external paths resolve from the file that declared the +surviving value, including imported fragments and configuration profiles. CLI +reference paths continue to resolve from the process working directory. + +Remove the lane-level `references` compatibility alias in version 5. New files +must place a lane's extraction references under `extract.references`. This +leaves one canonical location and removes merge-precedence behavior from the +file converter. + +### Strictness and dynamic options + +Keep ConfigKit's strict YAML rules. Version 5 therefore rejects aliases, merge +keys, unsupported tags and scalar forms, duplicate mapping keys, unknown typed +fields, trailing documents, and incompatible YAML types. Do not recreate an +application pre-parser to weaken these rules. + +Module `options` remain `map[string]any` where the selected module determines +the valid keys. ConfigKit represents YAML integers in dynamic values as +`int64`. Module option validators must accept that canonical representation, +check range before conversion, and hand runtime modules a safe `int` only where +their APIs require it. Do not marshal and decode dynamic subtrees through YAML +to recover native `int` values. + +## ConfigKit-Backed Loading Model + +Define a plain, typed configuration document for ConfigKit. It should mirror +the public file schema, use ordinary YAML tags, and contain no custom YAML +unmarshal hooks. Keep it separate from framework runtime types where doing so +avoids YAML concerns, presence flags, or configuration-only unions leaking into +`internal/framework/pipeline`. + +The reusable ConfigKit definition should provide: + +- fresh built-in defaults for each load; +- the `/version` rule; +- recorded whitespace normalization for identifiers, paths, profile IDs, + endpoints, and other fields whose current contract normalizes input; +- recorded derivation of `concurrency.stage_workers.extract` from the effective + `concurrency.total_llm` only when the worker value was omitted; +- collision checks before normalizing map keys or cross-step identifiers; +- configuration-only semantic validation with precise paths and related paths + where useful; and +- a dynamic secret predicate that covers credential-shaped keys below module + option maps, preserving the current defensive redaction boundary even though + ordinary Notarius configuration contains no credential values. + +Use ConfigKit's resolver only for deterministic configuration normalization and +derived defaults. Use its read-only semantic validator for domain constraints. +Do not put module lookup, PromptKit profile lookup, filesystem resource checks, +or runtime pipeline construction in either callback. + +After a successful load, perform one explicit, independently tested conversion +from the canonical configuration document to the Notarius runtime `Config` and +pipeline profile types. The converter should be a narrow ownership boundary, +not a second parser or validation framework. It may translate enums and checked +numeric values, but it must not silently change already validated meaning. + +Retain the ConfigKit result alongside the runtime configuration in an +application-owned loaded-configuration value. Commands that need only the +runtime `Config` should not gain access to parser internals; commands and debug +writers that expose safe configuration provenance may use the retained result. + +## Imports And Configuration Profiles + +Enable ConfigKit's explicit imports using its existing `configkit.imports` +syntax, including path, glob, and optional declarations. Operators can keep +whole pipeline definitions in separate files and compose them through the +selected root file. The root file alone declares `version`. + +Use ConfigKit's strict composition semantics. Imports combine disjoint +definitions; they do not provide ordered override layers or concatenate +sequences. In particular, one pipeline's ordered `steps` list remains together +in a single definition. Do not add Notarius-specific step merging or a second +import mechanism. + +Expose `--config-profile ` on every command that loads configuration: +`run`, `config validate`, `pipelines list`, `config effective`, and +`config explain`. Pass the selection to ConfigKit. A missing selection uses +the composed base configuration. ConfigKit's reserved selection `default` +also means no overlay; it does not load a default profile file. + +Use ConfigKit's profile location `/profiles/.yml` +and existing selection rules. A selected missing profile fails loading. +Profiles recursively overlay mappings and replace scalars, sequences, and +nulls as specified upstream. Development profiles can change pipeline-level +`llm_profile`, concurrency, and output settings without repeating the pipeline. +Changing an individual binding inside `steps` requires replacing the complete +steps sequence. Document this limitation prominently in examples. + +Configuration profiles and PromptKit profiles are independent. The former +select a configuration overlay with `--config-profile`; the latter select LLM +settings through `llm_profile` and the existing `--llm-profile` override. Neither +selection implicitly chooses the other. + +Use the selected file's canonical directory as ConfigKit's configuration root. +Document the upstream requirement that imports and configuration profiles stay +within that root, along with composition conflicts, cycle/repeated-import +errors, and profile sequence replacement. This containment rule does not apply +to Notarius input, reference, output, cache, debug, or PromptKit profile paths. + +### Paths And Source Ownership + +Resolve relative external reference paths and `promptkit.profile_file` and +`promptkit.profile_dir` relative to the file that supplied the surviving value. +This deliberately changes PromptKit profile-source paths from their current +working-directory convention in version 5. Absolute paths remain absolute. +Keep output, cache, and debug relative-directory semantics unchanged; they are +runtime destinations rather than files bundled with configuration fragments. + +Use ConfigKit's source metadata to identify each surviving declaration's +origin, including profile replacements and values inherited from the base. +Record path normalization so effective output and explanation reflect the +paths used at runtime. Preserve attribution when identifier normalization +renames an enclosing map key. Do not resolve overwritten or removed path +values or infer source ownership from diagnostic display labels. Explicit CLI +paths retain their working-directory interpretation. + +## Discovery, Precedence, And Overrides + +Keep the existing file-discovery order: explicit `--config`, then non-empty +`NOTARIUS_CONFIG`, then the installed default path. `NOTARIUS_CONFIG` selects a +source and is not a field-level configuration override, so it remains outside +ConfigKit. + +Retire these seven field-level environment overlays in version 5: + +- `NOTARIUS_TOTAL_LLM_CONCURRENCY`; +- `NOTARIUS_STAGE_WORKERS_EXTRACT`; +- `NOTARIUS_OUTPUT_DIR`; +- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`; +- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`; +- `NOTARIUS_CACHE_CHECKPOINTS_DIR`; and +- `NOTARIUS_DEBUG_DIR`. + +ConfigKit deliberately does not model general environment overlays. Applying +them after a successful load would make the retained effective configuration, +origins, and explanations disagree with the values used at runtime. Operators +must put durable values in YAML and use the supported explicit CLI flags for +invocation-local changes. + +Classify the current run flags by responsibility. A flag that directly +overrides a field in the typed configuration document should become a +ConfigKit `Override` record only when the operator explicitly supplied it; +parser defaults must not be submitted as overrides. Input selection, lane +selection, session identity, resume/recompute actions, command-line references, +and similar invocation state remain outside ConfigKit even when they affect +runtime resolution. The documented precedence becomes built-in defaults, +composed YAML sources, the selected configuration profile, explicit +configuration-valued CLI overrides, and then +invocation-only resolution actions at their existing application boundary. + +## Validation And Diagnostics + +Move configuration-only checks currently split among parsing, file +application, and `Config.Validate` into the ConfigKit +definition when they can be expressed against the typed file contract. Report +all independently discoverable semantic errors in one validation pass rather +than failing at the first field, while avoiding cascaded diagnostics whose only +cause is an earlier invalid parent. + +Keep catalog-dependent checks in `Config.Resolve` and +`pipeline.ResolvePipeline`. These include registered module and validator +existence, stage capabilities, artifact compatibility, reference-slot +compatibility, PromptKit profile availability, and the concrete resolved +pipeline digest. Their existing error classification and fail-before-prepare +behavior must remain intact. + +At the CLI boundary, recognize `*configkit.LoadError` and use ConfigKit's +canonical diagnostic renderer. Operators should receive the diagnostic code, +semantic path, safe message, and source coordinates when available. Do not +append raw YAML parser errors, source fragments, configuration values, or +secret values to the outward error chain. File discovery/read context should +remain clear without wrapping the same failure repeatedly. + +Nonfatal ConfigKit diagnostics, if Notarius defines any, must not be silently +dropped. They should enter the existing process-warning model only when they +represent an actionable configuration degradation. Ordinary normalization, +defaulting, and provenance records are explanation data, not run warnings. + +## Effective Configuration And Redaction + +Use `Result.WriteEffective` as the canonical renderer for the loaded effective +configuration. The requested debug bundle should store this deterministic, +redacted YAML rather than a separately maintained JSON clone of the raw +configuration. Update the artifact name and documentation together if the +format changes. + +Use ConfigKit's secret predicate to redact credential-shaped dynamic option +paths. PromptKit credentials remain environment references in PromptKit-owned +profile files and must not be copied into Notarius configuration or debug +state. + +Do not remove Notarius's redaction of the resolved pipeline merely because the +loaded configuration is ConfigKit-backed. The resolved value contains +application-owned structures, selection results, and reference provenance that +ConfigKit does not own; materialized reference bytes must continue to be +excluded from ordinary summaries. Remove only manual configuration cloning and +redaction made redundant by the frozen ConfigKit result. + +## Configuration Inspection Commands + +Add these read-only commands as first-class ConfigKit-backed inspection +surfaces: + +```text +notarius config effective [--config path/to/config.yml] [--config-profile name] +notarius config explain [--config path/to/config.yml] [--config-profile name] +``` + +`config effective` loads the selected configuration through the production +path and writes only ConfigKit's deterministic redacted YAML to standard +output. `config explain` accepts one canonical JSON Pointer, loads through the +same path, and writes ConfigKit's deterministic redacted explanation for that +effective or historical path. Neither command resolves a pipeline, prepares a +module, requires provider credentials, or contacts a provider. + +Both commands must use ordinary configuration discovery, ConfigKit diagnostic +rendering, the same import and profile selection as execution, and the existing +CLI exit-status conventions. An unknown or +noncanonical explanation path is a command failure, not an empty successful +result. Standard output contains only the requested inspection output; +diagnostics go to standard error. Effective output is YAML and explanation is +ConfigKit's human-readable text; neither is promised to be reloadable +configuration or a new durable JSON API. There is no unredacted mode. + +## Required Code And Contract Changes + +The implementation must address the following coherent work areas: + +- add and pin a reviewed `gitea.maximumdirect.net/eric/ConfigKit` release with + v0.2.0's semantics and the required source-origin API; +- introduce the version 5 typed document and its ConfigKit definition; +- replace `LoadFileConfig`, custom `UnmarshalYAML` implementations, manual + strict-parser checks, presence flags, `ApplyFileConfig`, and redundant + configuration cloning with the ConfigKit load boundary; +- consolidate defaults, normalization, and configuration-only validation into + the definition while retaining runtime catalog validation; +- update CLI configuration loading and error rendering without changing file + discovery; +- enable ConfigKit imports and configuration profiles, wire `--config-profile` + through all configuration-loading commands, and retain declaration origins + for relative file paths; +- retire field-level environment overlays while retaining `NOTARIUS_CONFIG`; +- classify current run flags so only true configuration overrides use + ConfigKit override records and invocation-only state remains outside the + configuration loader; +- update dynamic module-option decoding and range checks for `int64`; +- integrate ConfigKit effective rendering and metadata into requested debug + bundles and the new `config effective` and `config explain` commands; +- remove superseded code, tests, and the direct `gopkg.in/yaml.v3` dependency + after no production or test-only use remains; and +- update maintained examples and canonical documentation in the same change as + each externally visible contract adjustment. + +## Documentation And Migration + +Update `docs/config.md` as the canonical owner of version 5 syntax, defaults, +precedence, canonical bindings and references, strict YAML behavior, and the +retirement of field-level environment overrides. Replace the version 3 +migration section with a concise version 4-to-5 migration guide covering: + +- changing the numeric version value from 4 to 5; +- expanding scalar module bindings into `{module: ...}` objects; +- expanding scalar external references into `{path: ...}` objects; +- moving lane-level reference aliases to `extract.references`; +- resolving configured PromptKit profile-source paths from their declaring + file instead of the working directory; +- using imports for whole pipeline definitions and configuration profiles for + overlays, including complete replacement of step sequences; +- replacing retired operational environment overrides with YAML or supported + explicit CLI flags; and +- any effective/debug artifact or inspection-command changes. + +Update `docs/internal/configuration.md` to describe the ConfigKit boundary, +typed document-to-runtime conversion, application-owned validation, retained +metadata, and catalog-resolution handoff. Update `docs/cli.md` only for changed +commands or diagnostic behavior, `docs/operations.md` only for operator-facing +configuration and debug workflows, and the relevant architecture text if the +adopted precedence model changes. + +Convert both maintained D&D configurations and every configuration snippet to +version 5 canonical syntax. Examples must remain secret-free and load through +the same production path exercised by the CLI. Do not keep a second set of +version 4 examples. + +Keep the minimal D&D example self-contained. Extend the complete D&D example +with an imported whole pipeline definition and a selectable development +configuration profile. Keep configuration overlays physically distinct from +the maintained PromptKit profile files so readers can see which system owns +each file. Test the complete example both with and without the development +overlay, and update subprocess guidance to show optional `--config-profile` +selection. All configuration inspection commands must document what the +selected profile changes and that runtime pipeline resolution is separate. + +Because this change establishes a durable dependency and reallocates ownership +at a core boundary, add an ADR recording the decision to use ConfigKit, the +responsibilities retained by Notarius, the deliberate version 5 break, and the +retirement of the application environment-overlay layer. The rejected +alternatives should include continuing the custom loader, retaining version 4 +temporarily, or wrapping ConfigKit in a compatibility pre-parser. The ADR +should record rationale, not duplicate the current configuration reference or +implementation details. + +## Testing And Verification + +Follow the repository's behavior-focused testing policy. Reuse ConfigKit's own +tests for generic YAML and metadata mechanics; Notarius tests should protect +only its contract and integration responsibilities. + +Required Notarius coverage includes: + +- successful loading of omitted, explicit zero/false/empty, and derived values + that materially affect Notarius behavior; +- numeric and quoted version 5 acceptance and safe rejection of absent, null, + version 4, unsupported identifiers, and non-integer numeric forms; +- canonical binding and reference decoding plus rejection of former shorthand; +- identifier normalization and normalized-collision rejection; +- aggregation and source location of representative application validation + errors; +- unchanged configuration discovery and source-relative reference resolution; +- imported whole pipeline definitions, representative composition conflicts, + and profile precedence through the real ConfigKit loader; +- profile sequence replacement and explicit clearing, so Notarius does not + accidentally append or merge ordered steps; +- relative reference and PromptKit profile-source paths from root files, + imported fragments, and overlays, including an unchanged inherited value, + a replaced value, and an enclosing normalized identifier; +- matching configuration-profile selection across execution, validation, + pipeline listing, effective output, and explanation, without provider calls; +- checked dynamic integer conversion at each production module option boundary; +- detached ownership of loaded configuration and resolved pipelines; +- safe CLI rendering of representative ConfigKit parse, decode, version, + resolution, and validation failures; +- successful redacted `config effective` output and effective, historical, + unknown, and noncanonical `config explain` path behavior; +- effective-configuration redaction, including nested credential-shaped dynamic + option keys, without exposing materialized references; +- maintained minimal and complete example validation through the production + loader and production module catalog; and +- representative end-to-end CLI validation and pipeline resolution without + network access or credentials. + +Delete or rewrite legacy parser tests whose only purpose is to retest behavior +owned by ConfigKit. Retain a small integration sample for each strictness rule +that is itself part of the Notarius public contract, and avoid assertions on +incidental diagnostic prose. + +The completed implementation must pass: + +```sh +go test ./... +go vet ./... +go build ./cmd/notarius +``` + +It must also validate both maintained example files with the production CLI. +The complete example may use a placeholder credential for offline PromptKit +profile validation, as documented today; no live provider call is required. + +## Non-Goals + +- Moving pipeline resolution, module catalogs, or PromptKit profiles into + ConfigKit. +- Adding arbitrary environment-to-field binding. +- Retaining the retired field-level operational environment variables as a + parallel precedence layer. +- Loading credentials into the Notarius configuration document. +- Supporting remote configuration sources or formats other than YAML. +- Adding a generic dynamic-subtree conversion layer. +- Composing ordered steps across fragments or adding application-specific + sequence merge, profile inheritance, or profile stacking semantics. +- Preserving scalar shorthand with an application pre-parser. +- Changing the fixed pipeline shape or durable extraction artifacts. +- Changing run input, output publication, cache, checkpoint, retry, or warning + semantics except where configuration loading or debug rendering directly + exposes them. + +## Acceptance Criteria + +- All configuration-loading commands use a pinned ConfigKit release providing + the required source-origin API, building on v0.2.0, and no production code + directly imports a YAML parser. +- Version 5 is the only accepted file contract, documents `version: 5` as its + canonical spelling, and has one canonical object representation for module + bindings and references. +- The ConfigKit definition is the sole owner of typed defaults, input presence, + recorded normalization, derived configuration defaults, and + configuration-only semantic diagnostics. +- Notarius remains the sole owner of catalog-dependent resolution and runtime + conversion. +- Dynamic option integers are handled without unchecked narrowing or YAML + round-trips. +- CLI diagnostics and effective configuration output are deterministic, + redacted, value-safe, and retain useful path/source context. +- `config effective` and `config explain` expose only ConfigKit's redacted + inspection views and perform no pipeline or provider work. +- Imports compose whole pipeline definitions and selected configuration + profiles apply ConfigKit's documented overlay semantics consistently across + all configuration-loading commands. +- Relative external references and PromptKit profile-source paths use the + surviving declaration's source directory, and safe inspection agrees with + the paths used at runtime. +- The resolved pipeline remains detached, reference content remains excluded + from ordinary summaries, and its identity changes under the same semantic + conditions as before. +- Maintained examples, current documentation, focused tests, and repository-wide + validation agree with the implemented version 5 contract. +- `NOTARIUS_CONFIG` still participates in discovery, while the seven + field-level operational environment variables no longer affect loaded + configuration. +- Superseded custom parser, presence, environment, clone, redaction, and test + code is removed rather than left as an alternate loading path. + +## Deferred Composition Work + +Step-level composition may be revisited after operators use imported whole +pipeline definitions. It is not part of this adoption: ConfigKit does not +concatenate sequences or merge their elements by application-defined IDs. +Any future extension must define ordering, duplicate step IDs, dependency +validation, profile overrides, and provenance before changing that contract. diff --git a/docs/roadmap/deployment-managed-cache-permissions.md b/docs/roadmap/deployment-managed-cache-permissions.md deleted file mode 100644 index aaf2d486..00000000 --- a/docs/roadmap/deployment-managed-cache-permissions.md +++ /dev/null @@ -1,236 +0,0 @@ -# Deployment-Managed Cache Permissions - -Status: proposed - -## Summary - -Stop treating cache access permissions as application policy. Notarius should -create chunk-plan and checkpoint state through conventional Unix filesystem -semantics and leave the effective owner, group, and access policy to the -operator's provisioned directories, process umask, setgid inheritance, and -default ACLs. - -Notarius continues to own safe paths, symlink rejection, atomic publication, -cache validation, and recovery. It does not decide whether a configured cache -is private to one account, shared by a trusted group, or readable under some -other deployment policy. - -## Motivation And Current Problem - -The current cache implementations enforce a policy that was stronger than the -application requires: - -- the chunk-plan store creates and repeatedly converges cache directories to - `0700` and atomic files to `0600`; and -- the checkpoint recorder supplies `0700` directory and `0600` file modes to - the shared confined writer, which explicitly applies the file mode during - atomic publication. - -This prevents an administrator from using ordinary Unix provisioning to share -one cache between a service account and trusted operator accounts. Adding a -second Notarius-specific `private` or `group` configuration policy would solve -that immediate symptom, but it would preserve the underlying ownership error: -Notarius would still decide access policy instead of respecting the deployment. - -Routine Notarius operation does not require an application-enforced private -cache. Cache records can contain source-derived data, so operators must be told -what they are storing, but sensitivity does not imply that Notarius should -override a deliberately configured filesystem policy. API credentials remain -outside cache payloads and continue to follow their existing configuration and -redaction boundaries. - -## Target Policy - -The target contract is: - -> Notarius creates cache entries with conventional maximum creation modes and -> allows the operating system to determine effective access through the -> process umask, inherited group ownership, and default ACLs. - -Concretely: - -- newly created cache directories request `0777`; -- newly created cache files, including atomic temporary files, request `0666`; -- the kernel applies the invoking process's umask and any applicable default - ACL; -- setgid directories provisioned by the operator control inherited group - ownership and, on supported Unix systems, setgid propagation to descendants; -- Notarius does not call `chmod`, `chown`, or `chgrp` to impose or repair a - cache access policy; and -- Notarius does not reject a cache merely because its effective modes are more - or less permissive than a Notarius-selected default. - -This is policy inheritance, not literal copying of a parent directory's mode -bits. Directories and non-executable data files appropriately have different -maximum modes. A parent mode alone also does not determine child modes: an -operator who needs a particular result must configure the service umask and, -where appropriate, a setgid parent or default ACL. - -For example, a service started with umask `0077` ordinarily creates `0700` -directories and `0600` files. A trusted-group deployment can provision a -setgid cache root and use umask `0007`, or an equivalent default ACL, to obtain -`0770` directories and `0660` files. These are deployment choices rather than -configuration values interpreted by Notarius. - -## Public Contract - -No new Notarius configuration field, environment variable, or CLI flag is -introduced. In particular, do not add the previously considered -`cache.access: private|group` setting. - -Existing cache enablement, locations, chunk-plan modes, resume controls, -identity, and compatibility rules remain unchanged. The behavioral change is -that effective permissions on newly created cache entries are controlled by -the deployment environment rather than forcibly converged by Notarius. - -The policy applies uniformly to both cache families: - -- canonical chunk plans; and -- invocation checkpoints. - -Output and debug storage are outside this feature's scope. Their current -contracts must not change incidentally through a shared helper refactor. A -separate review may later consider whether the same deployment-owned policy is -appropriate for those surfaces. - -## Filesystem And Atomic-Publication Requirements - -Permission deference must cover every cache write path, not only final files: - -- configured cache-family roots created by Notarius; -- chunk-plan source-digest directories; -- checkpoint identity and stage directories; -- atomic temporary files; and -- final files published by rename. - -Atomic publication must not bypass the deployment policy. Go's -`os.CreateTemp` creates a file with `0600`, and applying `chmod(0666)` afterward -would bypass the process umask. Cache writers therefore need an atomic -temporary-file creation path that chooses a collision-resistant name and opens -it with `O_CREATE|O_EXCL` and requested mode `0666`, allowing the kernel to -apply umask and default ACL policy at creation time. - -Replacement writes publish a newly created temporary inode. Its effective -permissions and ACL are therefore those selected for new entries by the -current parent and process policy; Notarius does not copy mode bits or ACLs -from the replaced file. Operators who require stable replacement permissions -must express that policy through a stable service umask and directory default -ACL. - -Opening an existing cache for writing must not alter its root or descendant -modes. Read-only lookup of a missing cache must remain side-effect free. - -## Preserved Safety And Correctness Boundaries - -Deferring access policy does not weaken Notarius's filesystem correctness -rules. The implementation must preserve: - -- confined resolution beneath the selected cache root; -- rejection of symlinked or unexpected cache entries; -- validation of source digests, checkpoint identities, schemas, and recorded - provenance; -- atomic replacement, existing synchronization behavior, cleanup of abandoned - temporary files, and complete-record visibility; -- existing missing, malformed, incompatible, and operational-error recovery - classifications; and -- separation of output, cache, and debug roots. - -An administrator's decision to permit multiple accounts to reach one cache -does not create a new Notarius guarantee for simultaneous writers. Preserve -the existing atomicity and validation behavior, but do not add locking or claim -transactional coordination across concurrent processes as part of this work. -Concurrent writes to the same multi-file checkpoint identity remain subject to -the currently documented cache behavior; the implementation documentation -must state any existing limitation accurately. - -## Architectural Boundaries - -The target removes application-owned cache modes without creating a general -permission-policy framework. Chunk-plan and checkpoint storage may use -different filesystem APIs internally, but both must implement the same -deployment-owned result. Pipeline modules and the runner remain unaware of -physical roots and filesystem access policy. - -Any shared writer API must distinguish an application-enforced exact mode from -an umask-filtered maximum creation mode. The cache change must not silently -alter output or debug behavior merely because checkpoints currently share -filesystem helpers with those surfaces. - -## Existing Cache Migration - -Stopping permission convergence does not change existing inode modes. A cache -tree created by an older Notarius version may therefore remain `0700`/`0600` -and inaccessible to another account after upgrade. - -Operators who want a different policy must either: - -- remove the reconstructible cache and allow it to be recreated beneath a - correctly provisioned parent and service environment; or -- stop writers and deliberately update ownership, modes, and ACLs using normal - system-administration tools. - -Notarius must not perform a recursive startup migration. The same rule applies -when an operator wants to make a previously shared cache private: change the -deployment policy and deliberately migrate or recreate existing state when an -immediate whole-tree change is required. - -## Documentation And Decision Records - -The final documentation set includes an accepted ADR titled along the lines of -"Defer cache access policy to the deployment." The ADR records the durable -ownership boundary so later maintenance does not reintroduce -application-enforced modes solely because cache data can contain source-derived -content. - -The new ADR need not supersede ADR-0005 or ADR-0007: those decisions govern -canonical chunk-plan reuse and checkpoint recording/reuse, while leaving exact -storage permissions to operational contracts. It should reference those ADRs -and the output/cache/debug separation established by ADR-0006 and refined by -ADR-0007. - -In the final current-behavior documentation, `docs/operations.md` owns cache -provisioning, umask, setgid, default ACL, migration, sensitivity, and -concurrent-use guidance. Architecture owns the deployment/application boundary; -the configuration reference clarifies that selecting a cache path does not -select its permissions; and the internal state guide explains the implemented -write mechanics. The release that first contains the feature summarizes the -operational change without becoming its canonical reference. - -Current-behavior documentation must not describe this target behavior before -the implementation lands. - -## Verification Expectations - -Behavioral tests protect kernel-selected permissions for both cache families, -non-mutation of existing roots, atomic replacement, and representative -assembled CLI wiring. Existing confinement, symlink, validation, identity, -reuse, and recovery coverage remains authoritative. Umask tests are POSIX-only, -serialized, and restore process-global state. The default suite does not create -privileged accounts or groups; setgid and default-ACL examples may be verified -in a purpose-built Linux deployment environment. - -## Acceptance Criteria - -- No cache access-mode field or application-specific private/group policy is - added. -- Both chunk-plan and checkpoint writers request `0777` for new directories - and `0666` for new files, including atomic temporary files. -- Effective permissions reflect the process umask and applicable default ACLs - rather than a later Notarius chmod. -- Existing cache roots and descendants are not chmod-converged during reads or - writes. -- A private deployment can obtain private cache state through its filesystem - provisioning and umask without a Notarius-specific setting. -- A trusted-group deployment can obtain shared cache state through setgid, - umask, and/or default ACL policy without a Notarius-specific setting. -- Existing cache trees are not recursively migrated. -- Chunk-plan and checkpoint enablement, lookup, recording, resume, identity, - validation, confinement, atomicity, and recovery semantics remain unchanged. -- Output and debug permission behavior does not change as an incidental effect. -- Notarius performs no user/group management, ownership mutation, ACL - management, privilege escalation, or cross-process locking. -- The ADR and canonical operations, architecture, configuration, and internal - documentation accurately describe the implemented boundary; the next - release note summarizes the change when that release is prepared. -- Focused tests and repository-wide `go test ./...`, `go vet ./...`, and - `go build ./cmd/notarius` checks pass. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 958f8bc7..91f666eb 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -150,16 +150,6 @@ address Go dependency cycles, registrar ownership, stable public module keys, and supersession of the affected ADR-0004 decision. Conceptual artifact-family ownership does not by itself require this move. -## Near-Term Operations - -### Deployment-Managed Cache Permissions - -- Implement the proposed - [deployment-managed cache permissions](deployment-managed-cache-permissions.md) - contract so Notarius respects the access policy established by the cache - root, process umask, setgid inheritance, and default ACLs instead of imposing - application-owned private or group modes. - ## Blue-Sky Platform And Operations These ideas are intentionally less specified. Promote one into an earlier diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 6309d848..3513b43a 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,357 +1,700 @@ -# Deployment-Managed Cache Permissions Implementation Plan +# ConfigKit Adoption Implementation Plan -## Purpose +## Purpose And Execution Contract -Implement the target contract in -[Deployment-Managed Cache Permissions](deployment-managed-cache-permissions.md): -Notarius must preserve safe, confined, atomic cache I/O while allowing normal -Unix creation semantics to determine who can access newly created chunk-plan -and checkpoint state. +Implement the target in [ConfigKit Adoption](configkit.md). That document owns +the feature scope and operator choices; this document owns the implementation +sequence. Each numbered stage is one implementation prompt for a +gpt-5.6-terra coding agent. Execute stages in order and finish the named checks +before proceeding. Do not combine the configuration cutover with unrelated +pipeline or security refactoring. -This document is the execution sequence. The feature roadmap is authoritative -for user intent, policy, scope, and the target end state. Each numbered stage -below is sized for one `gpt-5.6-terra` implementation prompt and must be -completed in order. +Read `AGENTS.md`, `docs/development.md`, all three documents under +`docs/policy/`, and the relevant focused internal documentation before acting. +Apply the implementation-discipline policy: use the dependency's actual APIs, +preserve the requested flexibility, and do not add application restrictions +under the guise of this migration. Tests protect behavior, not exact helper +structure, incidental formatting, or a duplicated implementation of ConfigKit. -## Cross-Stage Requirements +The selected approach requires an upstream extension: ConfigKit v0.2.0 cannot +expose source origins during its resolution callback. See **Upstream +Prerequisite**. Stages 1–6 are independent of that missing capability. Stage 7 +and all later stages have an explicit gate; the sprint is not ready for +unattended execution through that gate until the upstream release is available. +Do not mark a gated stage complete by weakening the feature contract. -Every stage must preserve these decisions: +Temporary coexistence of the old production loader and the new loader during +these stages is implementation scaffolding only. There is no supported dual +version loader in the final product and no release should be tagged midway +through the cutover. Each stage must compile and pass its scoped tests; the +final stage establishes repository-wide readiness. -- Do not add a `cache.access` field, environment variable, CLI flag, or an - application-level private/group policy. -- New cache directories request mode `0777`; new cache files request mode - `0666`. The kernel applies umask and default ACL policy. -- Do not chmod, chown, chgrp, or recursively migrate existing cache entries to - enforce access policy. -- Keep chunk-plan and checkpoint confinement, symlink rejection, identity, - validation, atomic replacement, cleanup, and recovery behavior intact. -- Do not change output or debug permissions as an incidental consequence of a - shared-helper change. -- Do not add cross-process locking or promise transactional concurrent writes - to one checkpoint identity. -- Follow `docs/policy/testing.md`: protect the observable permission boundary - and meaningful filesystem risks without snapshotting helper structure or - duplicating all cases at every layer. -- Keep the repository buildable and the tests relevant to the completed stage - passing. Do not retire either roadmap document during implementation; they - remain review evidence until the completed feature is accepted. +## Repository Baseline -## Stage 1: Record The Decision And Add An Umask-Respecting Atomic Writer +Planning examined Notarius at `946fa10` plus the outstanding policy and roadmap +changes. The sibling ConfigKit checkout is tagged `v0.2.0`. -**Completed:** 2026-09-11 +Relevant current owners are: -### Goal +| Area | Repository evidence | +| --- | --- | +| Version 4 YAML and conversion | `internal/core/config/file_config.go`, including `FileConfig`, custom `UnmarshalYAML` methods, `ParseFileConfigYAML`, `LoadFileConfig`, and `ApplyFileConfig` | +| Defaults, runtime ownership, domain validation | `internal/core/config/config.go`, `validation.go`, `effective_config.go`, and their contract tests | +| Field-level environment overrides | `internal/core/config/env.go` and `env_contract_test.go` | +| Configuration discovery and command overrides | `internal/cli/run.go`: `loadConfig`, `runConfigValidate`, `runPipelinesList`, and `runPipelineCommand` | +| Catalog checks and configured/CLI reference behavior | `internal/framework/pipeline/profile.go`, `references.go`, and CLI reference contract tests | +| Dynamic integer consumers | `internal/modules/generic/chunk/units/chunker.go` and `internal/modules/generic/output/json/encoder.go` currently require native `int` | +| Debug configuration and resolved-pipeline output | `internal/core/config/redaction.go`, `internal/core/debugbundle/summary.go`, and the CLI debug handoff | +| Maintained examples | `examples/dnd-minimal.config.yml`, `examples/dnd-complete.config.yml`, `examples/profiles/dnd-extraction.yml`, and CLI production contract tests | +| Direct YAML test dependency | `internal/cli/production_contract_test.go` also parses embedded prompt definitions using `yaml.v3` | +| Upstream contract | ConfigKit `docs/config.md`, `docs/consumers/README.md`, `load.go`, `resolution.go`, `metadata.go`, `records.go`, `path.go`, and consumer examples at v0.2.0 | -Record the durable ownership boundary and add the narrow filesystem primitive -needed by checkpoint storage, without changing any production caller yet. +The sibling checkout is reference material, not a production dependency path. +Do not commit a local `replace`, modify upstream, or invent a future API as part +of these Notarius stages. -### Required Changes +## Decisions Applied Throughout -1. Add `docs/adr/0017-defer-cache-access-policy-to-deployment.md` using the - repository's Nygard ADR format, with status `Accepted` and the implementation - date. The decision must: - - assign cache access policy to deployment provisioning, process umask, - setgid inheritance, and default ACLs; - - distinguish access policy from Notarius-owned confinement, validation, - atomicity, and recovery; - - reject both forced private modes and a Notarius-specific private/group - setting; - - state that source-derived cache contents may be sensitive and that the - operator owns access and retention; - - state that API credentials must remain outside cache payloads under the - existing secret boundary; - - explain that existing cache modes are not migrated automatically; - - reference ADR-0005, ADR-0006, and ADR-0007 without superseding them; and - - record application-enforced `0700`/`0600`, configurable permission modes, - and recursive permission convergence as rejected alternatives. -2. Extend `internal/core/fileio` with exactly two cache-oriented entry points: - - `WriteBytesWithProcessPermissions(root, name string, data []byte) error`; - - `WriteJSONWithProcessPermissions(root, name string, payload any) error`. -3. The new byte writer must reuse `SafePath` and symlink-component rejection, - call `os.MkdirAll` with `0777`, and atomically replace the target. The JSON - wrapper must retain the existing indented JSON plus trailing-newline format. -4. Implement the new atomic path independently from the existing exact-mode - `WriteBytes` path: - - generate a 16-byte cryptographically random suffix; - - attempt at most 32 unique sibling names of the form - `..tmp-`; - - open with `os.O_WRONLY|os.O_CREATE|os.O_EXCL` and mode `0666` so the kernel - applies umask and default ACLs at creation; - - retry only name collisions; - - write, close, rename, and remove an unpublished temporary file on every - failure path; and - - do not call `chmod` or inspect/change the process umask in production code. -5. Leave `WriteBytes`, `WriteJSON`, and their existing callers unchanged so - output and debug behavior cannot drift in this stage. +### Configuration And Dependency Boundary -### Tests +- Initially pin `gitea.maximumdirect.net/eric/ConfigKit v0.2.0`. Stage 7 may + require a reviewed release providing the agreed upstream extension. A later + pin must be explicit, never an unreviewed `main` revision. +- Use a string-backed document version and a `/version` rule accepting exactly + text `5`; both numeric `version: 5` and quoted `version: "5"` are accepted. + Maintained examples use the numeric form. Reject missing/null/unsupported + versions. Numeric v4 fails the application checker with migration guidance. +- Use plain YAML-tagged document structs in `internal/core/config`, without + custom YAML hooks. Keep runtime framework types independent of ConfigKit. +- Keep the existing runtime `Config`, `ResolveInput`, and `EffectiveConfig` + boundary where it is useful. A new `LoadedConfig` owns the private ConfigKit + result and exposes runtime configuration plus safe writer methods. No + caller may mutate a returned `Result.Config` and expect its frozen inspection + output to change. +- ConfigKit owns composition, strict decoding, metadata, and safe rendering. + Notarius owns typed defaults, field semantics, recorded normalization, + conversion, catalog resolution, and physical run preparation. +- Application paths use typed ConfigKit key/index constructors; never build + JSON Pointers by concatenating unescaped pipeline IDs or reference keys. -- Add supported-Unix (`linux || darwin`) behavioral tests using - `syscall.Umask`. Each test must restore the prior umask with `defer` and must - not use `t.Parallel`. -- Prove a restrictive umask (`0077`) produces `0700` nested directories and a - `0600` final file. -- Prove a group-oriented umask (`0007`) produces `0770` nested directories and - a `0660` final file. -- Pre-create the selected root with a distinctive mode, write beneath it, and - prove the root mode is unchanged. -- Replace an existing file created under a different mode and prove the new - inode follows the current process policy and contains the complete new - content. -- Prove no temporary sibling remains after success. Retain all existing - confinement, symlink, and exact-mode writer tests. +### Document Shape And Presence -### Validation +Use a distinct finite `ValidatorDocument` containing only `module`, +`llm_profile`, `structured_output_repair_attempts`, `retries`, and `options`. +The producer `BindingDocument` contains those fields plus references, +validation policy, and a validator list. This avoids recursive Go schema types +and makes forbidden nested validator fields strict unknown-field errors. -Run: +Use pointer scalars only when nil means inheritance or omission, and ordinary +structs/maps/slices elsewhere as appropriate to the contract. Use ConfigKit +presence before normalization changes paths. Remove custom parser-presence +flags. In particular: + +- omitted validators use the production chain; explicit `[]` selects an empty + chain; present `null` is invalid, matching the existing array contract; +- omitted repair attempts inherit; explicit zero disables repair; present + null is invalid for this integer setting; +- omitted LLM-profile fields inherit; explicitly blank or null selections are + invalid; +- omitted optional module bindings permit existing catalog defaults; a present + binding must be a valid object with a nonempty module; +- reference objects have exactly one selected source: nonempty `path` or a + complete `artifact` object with step and lane; explicit invalid branches must + not disappear during pointer conversion; +- preserve the single-step `artifacts` form and the ordered `steps` form; + reject simultaneous declarations and explicitly empty/null `steps`; +- retain optional null behavior where the existing contract permits clearing + an optional object or collection; do not impose a universal no-null rule; +- remove the lane-level reference alias. Keep the existing treatment of the + deprecated lane-level validator list during migration (empty allowed, + nonempty rejected) rather than adding an unrelated contract removal. + +Static defaults stay as currently documented. Derive omitted extraction workers +from the final configured total LLM limit. ConfigKit profile null replacement +is subject to the destination field's existing domain validity; overlay support +does not make null valid for every setting. + +### CLI And Path Ownership + +| Input | Owner / treatment | +| --- | --- | +| `--config`, `NOTARIUS_CONFIG`, installed fallback | Existing CLI discovery; ConfigKit receives the selected file | +| `--config-profile` | ConfigKit profile selection on all five configuration-loading command forms | +| Explicit `--output-dir` | ConfigKit override at `/output/directory` | +| Explicit `--debug-dir` | ConfigKit override at `/debug/directory`; preserve `--debug` requirement | +| Explicit `--chunk_cache` | ConfigKit override at `/cache/chunk_plans/mode` | +| `--llm-profile` | Existing runtime selection overriding all selected LLM bindings; not a replacement for a single YAML field | +| Input, lanes, session ID, reasoning effort, references, resume/recompute, debug enablement, JSON output | Existing invocation actions outside ConfigKit | +| Seven retired operational environment variables | No longer read or applied; no deprecation-warning scan of process environment | +| PromptKit credential environment variables | Existing PromptKit responsibility | + +Only explicitly supplied flags create override records, with safe flag names +as provenance. Preserve current blank-value checks and invocation error classes. +For `--config-profile`, pass the supplied name through ConfigKit's selection +rules; empty and exact `default` select no overlay. Do not invent stricter name +validation or whitespace normalization than the upstream profile contract. + +Enable imports without a separate Notarius enablement flag. The canonical +selected file's directory is the configuration root. Relative file references +and PromptKit profile sources use the surviving declaration's directory; +absolute values remain absolute. Resolve these to absolute paths before +runtime conversion, without reading the referenced contents. Output/cache/debug +relative directories retain working-directory semantics. CLI reference paths +retain the existing working-directory semantics and generated-reference +protection. Configuration-root confinement applies only to ConfigKit's imported +configuration and overlay files. + +## Stage 1 — Add ConfigKit And The Version 5 Document Types + +**Goal:** Establish the new file contract without switching production commands. + +**Work:** + +1. Pin v0.2.0 in `go.mod`/`go.sum` and inspect its actual exported API. +2. Add `document.go` and `load.go` in `internal/core/config` (or equivalent + cohesive files) with the complete plain document types, finite validator + type, explicit reference union, and YAML-tagged validation-policy fields. +3. Provide a private reusable definition factory with fresh static defaults and + the exact version rule. Define a small application `LoadOptions` for profile + selection and explicit configuration overrides; avoid a generic loader + abstraction or exposing parser types. +4. Keep the new load entry point unconnected to production until its + normalization and validation stages are complete. Mark it internal + scaffolding; do not advertise partially implemented v5 support. +5. Add the next available ADR (expected `0018-adopt-configkit.md`) recording + the accepted ownership, v5 break, imports/profiles, and environment policy. + Record the source-origin dependency without claiming it is implemented. + +**Acceptance:** v5 numeric/quoted versions decode; unsupported versions fail +before document decoding; representative canonical bindings and references +decode while shorthand/unknown fields fail. Separate loads do not alias +defaults. The current CLI still works through its existing loader. + +**Validation:** `go test ./internal/core/config`; `go test ./...` to confirm the +new dependency and schema types do not break the existing build. Keep tests +small; generic YAML exhaustiveness remains upstream. + +**Prompt size:** Small to medium; types, dependency, and version boundary only. + +## Stage 2 — Implement Operational Defaults And Normalization + +**Goal:** Make operational configuration behavior correct and inspectable. + +**Work:** + +1. Express existing concurrency, output, cache, debug, and PromptKit-local-backend + defaults and validations through the definition callbacks. +2. Use recorded `Default`/`Normalize` operations for derived extraction workers, + path whitespace, cache mode, and endpoint whitespace. Derived defaults must + follow profile/CLI overlays and preserve explicit zero for validation. +3. Use presence-aware checks for explicit empty/null PromptKit source selectors + and required fields in a present local-backend object. Preserve mutually + exclusive profile sources and the existing endpoint/range constraints. +4. Keep filesystem checks and PromptKit profile catalog lookup out of callbacks. + This stage trims profile-source paths but does not yet resolve their bases. +5. Extract small reusable value checks where needed so runtime validation and + document validation do not separately implement the same rule. Collect + independent errors through ConfigKit with safe messages and field paths. + +**Acceptance:** defaults and explicit values retain their meanings; worker +derivation follows effective total concurrency; multiple independent field +errors identify their source locations; normalization appears in explanation +without generating warnings. + +**Validation:** `go test ./internal/core/config`. Add load-boundary cases for +profiles overriding concurrency, CLI overrides, invalid output versus empty +cache paths, and local backend presence. No live profile resolution. + +**Prompt size:** Medium; operational fields only. + +## Stage 3 — Implement Pipeline Document Rules + +**Goal:** Validate and normalize the complete pipeline file shape before +catalog resolution. + +**Work:** + +1. Implement one typed traversal of pipelines, steps, lanes, bindings, + validators, and reference sources. Prefer shared binding helpers and + explicit walks over reflective generic validation. +2. Perform all presence-sensitive checks against original input paths before + renaming map keys. Detect whitespace collisions deterministically before + writing normalized keys. Report both paths where useful. +3. Normalize identifiers, selected profiles, and generated selectors with + recorded changes. For map-key moves, record the parent operation and relevant + source dependencies; preserve a load-local original-path mapping for Stage + 7. Do not guess origin from a normalized key after it has moved. +4. Enforce the field and shape contracts above, including global lane-ID + uniqueness across steps, valid retry values, validation-policy fields, and + allowed binding-local reference/validator placements. +5. Keep stage/module compatibility and actual option meaning in the existing + catalog resolver. Do not reject a module based on an embedded allowlist in + the document schema. +6. Factor shared semantic value checks with `Config.Validate` where necessary. + Keep file presence/shape checks in the document callback and direct runtime + construction checks at the runtime boundary; avoid parallel complete + validators with independently copied rules. + +**Acceptance:** single-step and ordered pipelines behave as specified; +explicit empty validator chains and zero repair budgets survive; invalid +references, profile selectors, normalized collisions, and duplicate lane IDs +fail with useful paths. No mutation is hidden from ConfigKit's change recorder. + +**Validation:** `go test ./internal/core/config ./internal/framework/pipeline`. +Migrate meaningful rule cases from the old parser tests to the new load +boundary as they gain an owner; retain old-loader coverage until cutover. + +**Prompt size:** Medium to large, bounded to pipeline document semantics; do +not combine it with source-origin or CLI work. + +## Stage 4 — Convert Documents Into Runtime Configuration + +**Goal:** Produce the existing typed runtime composition from the new document. + +**Work:** + +1. Add a narrow explicit conversion after successful ConfigKit loading. Build + independently owned `Config` and `pipeline.PipelineProfile` values, preserving + map identities, step order, inheritance pointers, empty chains, and reference + source forms. Convert the finite validator document to runtime bindings. +2. Wrap the result in `LoadedConfig`, retaining the private ConfigKit snapshot. + Provide safe effective/explanation writers and the runtime configuration; + avoid a public unredacted serialization method. +3. Stop re-deriving workers on the already finalized new-load path. Retain + programmatic `Config` validation and ownership checks. Any legacy + recomputation shim needed by the old loader is temporary and removed at + Stage 12. Runtime callers needing defaults should use the same default values + as the document definition, not a second literal default table. +4. Keep `Config.Resolve`/`pipeline.ResolvePipeline` responsible for selection, + capabilities, concrete validation-policy inheritance, options, references, + and digest generation. Do not add configuration profile name or source + history to semantic pipeline identity. + +**Acceptance:** equivalent canonical documents produce equivalent runtime +pipelines; changing a semantic setting changes identity, while merely importing +an equivalent definition does not. Mutating caller-owned maps/slices cannot +alter the snapshot or resolved pipeline. ConfigKit changes do not force its +types into module interfaces. + +**Validation:** `go test ./internal/core/config ./internal/framework/pipeline`. +Use a real test catalog and representative reference/validator chains; resolve +absolute reference fixtures until Stage 7 supplies relative path resolution. + +**Prompt size:** Medium; conversion and runtime ownership only. + +## Stage 5 — Support ConfigKit Dynamic Integers In Modules + +**Goal:** Ensure the module boundary accepts valid ConfigKit options. + +**Work:** + +1. Inspect every production numeric `map[string]any` option consumer. The known + sites are generic unit chunking and JSON evidence `window_units`. +2. Add one narrow shared option conversion helper in the existing pipeline + option-helper area. Accept `int64` from ConfigKit and native `int` from + programmatically constructed module bindings. Range-check `int64` before + conversion; keep domain positivity/nonnegativity at the module owner. +3. Preserve rejection of floats and strings. Do not recursively rewrite all + options, change their serialized representation, or add general numeric + coercion. Keep options in ConfigKit form through config conversion. +4. Update the old test that rejects `int64` to protect the newly accepted + boundary. Add representative real-loader-to-module tests, including nested + JSON evidence options. + +**Acceptance:** integral options work from YAML and programmatic bindings; +fractional/string inputs and invalid ranges still fail before preparation. +No unchecked narrowing or YAML round-trip remains at these consumers. + +**Validation:** `go test ./internal/modules/generic/... ./internal/framework/pipeline ./internal/core/config`. + +**Prompt size:** Small; two known consumers and a narrow shared helper. + +## Stage 6 — Prepare Debug Rendering For The ConfigKit Result + +**Goal:** Prepare the publication boundary before changing command loading. + +**Work:** + +1. Add a small writer interface such as `WriteEffective(io.Writer) error` to + the debug summary boundary. `LoadedConfig` satisfies it without exposing a + concrete ConfigKit result to debug storage. +2. Add publication of `effective-config.yml` through the existing confined, + atomic summary file writer. Buffer redacted output before publishing if the + current file API takes bytes; propagate rendering and write errors. +3. Preserve the existing redacted resolved-pipeline writer and all reference + content exclusion. The old `effective-config.json` writer remains only until + the run command switches in Stage 10. +4. Reuse `sensitiveConfigKey` for the definition's dynamic secret predicate at + nested options paths, including validator options, containers, and moved + map keys. Keep scalar classification policy in one owner. Do not log raw + configuration or metadata snapshots that contain values. + +**Acceptance:** redacted effective YAML publishes atomically; failure leaves no +partial artifact; nested sensitive values never appear in effective output or +explanation. Resolved runtime summaries still omit materialized references. + +**Validation:** `go test ./internal/core/config ./internal/core/debugbundle`. +Test rendering failure and caller mutation at these real local boundaries. + +**Prompt size:** Small to medium; rendering interface and publication only. + +## Stage 7 — Resolve Declaration-Relative Paths + +**Goal:** Normalize file paths using their effective source origins and retain +correct ConfigKit inspection history. + +**Prerequisite gate:** A reviewed ConfigKit release must expose surviving input origin and canonical +source information during resolution. Confirm the actual API and pin that +release before implementing this stage. If only v0.2.0 is available, stop here +and report the unmet dependency. Do not implement an upstream fork, reload the +configuration twice, parse YAML independently, or silently change the promised +effective output. + +**Work after the gate:** + +1. Resolve path origins at original input paths before map-key normalization + and before recording changes can replace a source origin. Use the scalar + origin and source ID to get the canonical declaring file; do not pick the + first origin of a composite parent mapping. +2. Join nonempty relative external reference paths and PromptKit profile-source + paths to that file's directory. Preserve absolute values. Do not read or + require existence of referenced files in this callback. +3. Record normalized paths and map moves so the frozen effective view matches + runtime values and explanation can trace back to the original declaration. + Keep map collision checks ahead of mutation; process stable sorted keys. +4. Leave destination directories unchanged except existing whitespace + normalization. CLI reference paths remain runtime-owned. No configuration + root confinement or symlink policy is added to referenced application files. +5. Enable imports and selected profiles on the public new loader, using + `File` and the selected file's canonical directory as the root. Use + ConfigKit's composition and profile behavior directly. + +**Acceptance:** root, fragment, and overlay values resolve from their own files; +inherited values retain their base origin; profile-replaced sequences use the +new declarations; normalized enclosing IDs do not lose attribution. Effective +and explanation output agree with the actual paths. Removed/overridden file +values are not opened or resolved. Root containment affects only ConfigKit +configuration sources. + +**Validation:** `go test ./internal/core/config ./internal/framework/pipeline`. +Use temporary configuration trees with same-named reference files in different +directories, aliases/symlinks accepted by ConfigKit, profile replacements, +normalized keys, and absolute external paths outside the config root. Use real +reference materialization for one case to prove the intended bytes are chosen. + +**Prompt size:** Medium, with a firm upstream gate; no CLI or example migration. + +## Stage 8 — Prepare Shared CLI Loading And Diagnostics + +**Goal:** Make one command-facing configuration entry point ready for cutover. + +**Work:** + +1. Add a CLI helper that discovers the file, passes profile selection and + explicit override records to the new loader, and returns `LoadedConfig` + plus the selected config path. Keep discovery order and injected test lookup. +2. Add a shared load-error reporter using `errors.As` for ConfigKit load errors + and `WriteDiagnostics`; supply safe source context using the dependency's + source records. Do not flatten multi-error output into repeated wrappers or + print raw parser/source contents. Preserve status 1 for valid invocations + whose configuration cannot load. +3. Add shared flag registration/parsing only where it reduces actual repeated + code. Use existing CLI parsing conventions, including flags surrounding + positional arguments. Preserve missing-value and unexpected-argument checks. +4. Route successful ConfigKit process diagnostics through one reporting owner. + Define no new application warnings for successful imports, defaulting, + normalization, or ignored retired environment names. Current application + rules should ordinarily return no nonfatal configuration diagnostics. + +**Acceptance:** the helper returns a fully validated loaded value with safe +diagnostics and uses the requested source/profile. No production command has +switched accidentally; existing command tests remain green. + +**Validation:** `go test ./internal/cli ./internal/core/config`. Add direct +helper tests for explicit flags versus parser defaults, discovery, safe +multi-error reporting, and profile selection. Use the real ConfigKit boundary. + +**Prompt size:** Medium; helper and diagnostic wiring only. + +## Stage 9 — Prepare Validation And Pipeline Listing Commands + +**Goal:** Finish the read-only command implementations before the shared +production cutover. + +**Work:** + +1. Implement v5 handlers for `config validate` and `pipelines list` using the + new helper and `--config-profile`. Preserve `--pipeline`, `--only`, JSON + listing, sorting, and exit/stream behavior. Keep production dispatch on the + old handlers until Stage 10 so all configuration commands switch together. + Use temporary explicit handler names, not a runtime version switch. +2. Plain validation is configuration-only. Selected-pipeline validation still + performs catalog and explicit PromptKit profile checks at their existing + owners, without provider calls or credential requirements beyond existing + offline profile validation. +3. Test these handlers with dedicated v5 fixtures and production registries. + Keep existing dispatch and maintained-example tests intact until Stage 10. + Do not weaken tests to accept both versions or advertise the unconnected + handlers as implemented CLI features. + +**Acceptance:** the new handlers see imported pipelines and overlay changes, +reject v4, and retain useful module/profile failure context. Existing public +command behavior stays coherent until all three handlers switch in Stage 10. + +**Validation:** `go test ./internal/cli ./internal/core/config`; run the focused +production validation cases using v5 fixtures, including unknown modules and +incompatible generated references. + +**Prompt size:** Medium; two closely related read-only commands. + +## Stage 10 — Switch Run And The Maintained V5 Configurations + +**Goal:** Complete production cutover and make all runs use one ConfigKit load. + +**Work:** + +1. Switch `runPipelineCommand` to the new loader. Translate only the three + config-valued flags in the decision table to ConfigKit overrides; remove + their direct post-load mutations. Add `--config-profile` consistently to + argument reordering and missing-value checks. Activate Stage 9's validation + and listing handlers in the same change and remove superseded handlers. +2. Preserve runtime LLM override, references, input, session identity, + resume/recompute, scheduler setup, and output receipt semantics. Config + failures must still occur before source parsing and physical run allocation. +3. Pass the retained loaded result to the effective YAML debug writer. Continue + to write the separately redacted resolved pipeline after reference + materialization. Add effective configuration profile selection to debug + invocation metadata (`config_profile`), with no overlay represented as empty. + Do not add profile-selection text to checkpoint or pipeline identity. +4. Convert both maintained D&D example configurations to v5 object syntax in + place, still as self-contained files in this stage. Change the complete + example's PromptKit source path for declaration-relative interpretation. +5. Migrate run/reference/state tests and the maintained-example loader to v5. + Split mechanical fixture updates from behavioral assertions within the + diff so regressions remain reviewable. Retired env-based tests now supply + YAML or explicit flags; keep one test proving those variables have no effect. +6. Update current CLI, configuration, operations, and consumer references for + v5, profile selection, environment retirement, and `effective-config.yml`. + +**Acceptance:** all production commands use the new helper; imported/profile +configuration and explicit overrides agree with execution and debug output; +the maintained minimal and full fake-provider workflows still complete with +the same artifact families and valid reference handoffs. + +**Validation:** `go test ./internal/cli ./internal/core/config ./internal/core/debugbundle ./internal/framework/pipeline`. +Use the existing complete D&D fake LLM test instead of a live backend. Verify +JSON run receipt and stderr warning behavior remain unchanged. + +**Prompt size:** Largest cutover prompt, limited to wiring, fixture migration, +and its current documentation. Schema, paths, numeric adapters, diagnostic +helpers, and debug writer must already be complete in earlier stages. + +## Stage 11 — Add Effective And Explanation Commands + +**Goal:** Expose the accepted operator inspection workflow. + +**Work:** + +1. Add `config effective` and `config explain` with the syntax in the feature + roadmap, including `--config` and `--config-profile`. +2. Effective writes only `LoadedConfig.WriteEffective` output. Explain accepts + exactly one JSON Pointer positional argument, parses it with ConfigKit, and + calls the safe explanation writer. Accept the empty pointer for the root + when explicitly supplied; reject malformed escapes/argument shape with + status 2. A well-formed unknown path returns status 1 and no success output. +3. Use ConfigKit's contextual sequence-index rules. Signed numeric mapping + keys remain valid keys; `/steps/+0` cannot identify a sequence element. + Do not add a blanket numeric-token restriction that rejects legal map keys. +4. Preserve writer failure handling. Explanations are human-oriented text; + effective configuration is YAML. Neither output is promised to be a + reloadable config or a new durable JSON API. Do not print success banners. +5. Avoid LLM client construction, module preparation, reference materialization, + state allocation, or pipeline selection. Existing catalog registration at + CLI composition is acceptable; it must not trigger provider work. +6. Update `docs/cli.md`, the configuration inspection guidance, and help. + +**Acceptance:** both commands work against imports/overlays, redaction applies +to history and effective values, removed keys remain explainable, malformed +and unknown paths fail appropriately, and stdout contains only requested data. + +**Validation:** `go test ./internal/cli ./internal/core/config`. Use real +ConfigKit with temporary files and a provider factory that fails if called. + +**Prompt size:** Medium; two commands sharing the completed load boundary. + +## Stage 12 — Remove The Legacy Configuration Machinery + +**Goal:** Finish the simplification promised by the adoption. + +**Work:** + +1. Remove the unused v4 parser, YAML hooks, `FileConfig` application APIs, + environment-overlay code, parser-only presence flags, and old loading helper. +2. Remove `extractWorkersConfigured`/`defaultedExtractWorkers` inference once + file defaults are finalized by ConfigKit. Programmatically constructed + configs treat a supplied worker entry as explicit and derive only an absent + entry; update internal construction tests accordingly. Keep runtime checks + through `Config.Resolve` for direct callers, without mutating shared maps. +3. Delete the superseded `effective-config.json` publication method and raw + configuration summary/redaction functions. Retain runtime ownership cloning + and resolved-pipeline redaction that still serve real callers. +4. Remove obsolete parser/env tests; retain behavioral coverage at the new + owner. Remove `yaml.v3` from the embedded-prompt test in + `production_contract_test.go`: exercise the production PromptKit loader or + existing asset-loading seam, rather than adopting ConfigKit to parse + PromptKit assets or adding another YAML dependency for the test. +5. Run `go mod tidy`; remove the direct `gopkg.in/yaml.v3` requirement and verify + Notarius Go files no longer import a YAML parser. Dependencies may retain + their private YAML implementation. + +**Acceptance:** no alternate loader, environment application layer, redundant +configuration redactor, or parser-only clone remains. Legitimate pipeline and +artifact serialization code is unaffected. Programmatic config behavior has +an explicit tested ownership/default contract. + +**Validation:** `go test ./...`; `go vet ./...`; search for `UnmarshalYAML`, +`ApplyFileConfig`, `ApplyEnvOverrides`, `gopkg.in/yaml`, and old effective-config +artifact references. Inspect matches rather than treating historical docs or +negative migration tests as failures. + +**Prompt size:** Medium; deletion and ownership consolidation after all callers +have moved. + +## Stage 13 — Demonstrate Imports And Development Profiles + +**Goal:** Supply two maintained operator examples covering simple and composed +configuration, with a clear distinction between the two profile systems. + +**Work:** + +1. Keep `examples/dnd-minimal.config.yml` self-contained. +2. Keep `examples/dnd-complete.config.yml` as the complete root and move its + whole `dnd-session` pipeline into + `examples/pipelines/dnd-session.yml`, imported through `configkit.imports`. + Only the root declares version. Fix fragment-relative campaign/catalog + references to `../dnd-party.txt`, `../dnd-glossary.txt`, and + `../dnd-spell-catalog.json` where applicable. +3. Move the existing PromptKit example to + `examples/llm-profiles/dnd-extraction.yml` and point the root's + `promptkit.profile_file` to `./llm-profiles/dnd-extraction.yml`. +4. Add `examples/profiles/development.yml` as a ConfigKit overlay. Set + `concurrency.total_llm: 1` and `stage_workers.extract: 1`, plus a development + output directory. Include a short comment explaining how an operator can + change the pipeline's `llm_profile` independently; do not invent an + unavailable model profile just to demonstrate the field. The overlay has + no version or import directive. +5. Extend the existing maintained-example test to load base and development + configurations through production code, validate catalogs/profiles offline, + and exercise complete fake-provider reference materialization from a + working directory other than the example directory. +6. Update `docs/config.md`, `docs/operations.md`, `docs/consumers/subprocess.md`, + and `docs/consumers/dnd-pipeline.md` at their canonical scopes. Link the + examples; explain disjoint composition and whole-sequence replacement + instead of duplicating complete configs in prose. + +**Acceptance:** all example paths resolve, the base and development selections +validate, the full pipeline still contains every implemented D&D lane, and +operators can distinguish `--config-profile` from `--llm-profile`. + +**Validation:** `go test ./internal/cli`; production CLI validation for both +roots and the development overlay. No API key or live generation is needed; +use only the documented placeholder if offline PromptKit validation requires +one. Verify all moved example links. + +**Prompt size:** Medium; examples and their operator-facing documentation. + +## Stage 14 — Final Architecture And Contract Verification + +**Goal:** Verify the completed architecture and reconcile canonical docs with +the implemented boundaries. + +**Work:** + +1. Review the complete diff against `configkit.md` and this plan. Check that no + stage was bypassed, no source-origin workaround weakened the selected + contract, and no unsupported step-composition semantics slipped in. +2. Finalize `docs/internal/configuration.md`, `docs/internal/cli.md`, + `docs/internal/state.md`, and `docs/internal/overview.md`. Add + `docs/integrations/pkg-configkit.md` for the pinned API boundary and relevant + upstream semantics; link upstream for general library behavior. +3. Update architecture's configuration ownership/precedence text and finalize + the adoption ADR. Document public behavior in its canonical owner; do not + retroactively edit historical release notes or prior accepted ADR decisions. +4. Ensure ConfigKit warning/diagnostic handling, safe rendering, source-relative + paths, runtime option validation, direct construction, and digest semantics + all have sufficient focused coverage. Do not duplicate upstream's test + suite or create exact-output change detectors. +5. Verify current docs no longer recommend v4 syntax, retired environment + overlays, the former PromptKit example path, or `effective-config.json`. + Migration explanations and historical records may legitimately name them. +6. Complete the checks below and report any remaining issue explicitly. Do not + tag a release, push, or retire the active roadmap during implementation + without a separate request. + +**Acceptance:** every feature acceptance criterion holds, current docs and +examples agree with runtime behavior, all required checks pass, and the +remaining configuration code expresses Notarius policy rather than rebuilding +ConfigKit machinery. + +**Validation:** ```sh -gofmt -w internal/core/fileio/*.go -go test ./internal/core/fileio -go test -race ./internal/core/fileio -git diff --check -``` - -### Stage Acceptance - -The ADR records the agreed boundary; the new writer demonstrates both target -umasks without chmod; existing file-writing behavior has not changed; and the -stage is small enough for one implementation prompt. - -## Stage 2: Convert Canonical Chunk-Plan Storage - -**Completed:** 2026-09-11 - -### Goal - -Make every chunk-plan creation and replacement path follow deployment-managed -permissions while preserving the rooted filesystem and cache-recovery -protocol. - -### Required Changes - -1. In `internal/framework/chunkplan`: - - request `0777` when creating the configured root; - - remove the chmod of an opened existing root; - - request `0777` when creating a source-digest directory; - - remove the chmod branch for an opened existing digest directory; - - retain the before/open/after identity checks that defend against entry - replacement races; - - change the existing rooted `O_CREATE|O_EXCL` temporary-file open mode from - `0600` to `0666`; and - - remove the subsequent `temp.Chmod(0600)` call while preserving write, - sync, close, rename, cleanup, and interruption hooks. -2. Do not replace `os.Root` operations with ordinary path operations, weaken - unexpected-entry handling, or create a missing root during read-only load. -3. Rewrite `TestFilesystemStorePermissions` into a behavioral Unix test that - controls and restores umask. Consolidate permission cases rather than - retaining a stale forced-private assertion alongside the new contract. - -### Tests - -- Under umask `0077`, prove the root, digest directory, and published plan are - `0700`, `0700`, and `0600`. -- Under umask `0007`, prove they are `0770`, `0770`, and `0660`. -- Because the published plan is the renamed temporary inode, its final mode - proves the temporary-file creation policy; use the existing pre-rename hook - only if a separate assertion is needed to prevent a plausible regression. -- Pre-create both the root and digest directory with deliberate modes, save a - plan, and prove neither directory is modified. -- Retain the existing missing-root side-effect, symlink, entry-replacement, - interruption, atomic-replacement, record-validation, and recovery tests. - -### Validation - -Run: - -```sh -gofmt -w internal/framework/chunkplan/*.go -go test ./internal/framework/chunkplan -go test -race ./internal/framework/chunkplan -git diff --check -``` - -### Stage Acceptance - -Chunk-plan writes honor both controlled deployment policies, existing -directories are not converged, and all rooted atomicity and recovery coverage -passes. This stage is small enough for one implementation prompt. - -## Stage 3: Convert Checkpoints And Prove Assembled Cache Behavior - -**Completed:** 2026-09-11 - -### Goal - -Move checkpoints to the new shared writer, remove stale private-cache -expectations at the CLI boundary, and prove one assembled run applies the same -deployment policy to both cache families. - -### Required Changes - -1. Change the checkpoint recorder's JSON write boundary in - `internal/framework/checkpoint` to call - `fileio.WriteJSONWithProcessPermissions`. Do not thread modes through the - recorder or introduce a cache permission configuration value. -2. Replace the checkpoint round-trip test's forced `0700`/`0600` subsection - with focused POSIX permission coverage. Use a cache root that does not exist - before the recorder writes so root creation is covered. -3. Preserve checkpoint JSON representation, path layout, manifests, payloads, - identity, recording enablement, reuse eligibility, loader decisions, and - corrupt/incompatible-state behavior. -4. In `internal/cli/state_hardening_test.go`, stop applying - `assertRestrictedTree` to checkpoints. Retain that helper and assertion for - debug state, whose permission contract is unchanged. -5. Add one supported-Unix CLI integration test using the existing offline - state-test harness: - - configure both chunk-plan caching and checkpoint recording beneath - previously absent roots; - - set and reliably restore umask `0007` around the run; - - prove representative directories and final files in both cache families - are group-accessible (`0770`/`0660`); - - prove the run completes and the stored state remains reusable; and - - do not duplicate every package-level permission or recovery case. -6. Keep output and debug mode assertions unchanged and confirm that no - `cache.access` parsing or composition code exists. - -### Tests - -- Add checkpoint-level cases for umask `0077` and `0007`, covering - representative nested directories and both manifest and payload files. -- Prove a pre-existing checkpoint root is not chmod modified. -- Retain the existing all-stage round trip as the owner of serialization and - loader behavior rather than cloning it into each permission case. -- Ensure every umask-changing test is non-parallel and restores global state. - -### Validation - -Run: - -```sh -gofmt -w internal/framework/checkpoint/*.go internal/cli/*.go -go test ./internal/framework/checkpoint ./internal/cli -go test -race ./internal/framework/checkpoint ./internal/cli -go test ./internal/framework/chunkplan ./internal/core/fileio -git diff --check -``` - -### Stage Acceptance - -Both cache families follow process/filesystem policy through the real CLI -composition, checkpoint reuse remains intact, and output/debug permission -contracts have not changed. This stage is small enough for one implementation -prompt. - -## Stage 4: Update Canonical Documentation - -**Completed:** 2026-09-11 - -### Goal - -Make the implemented behavior discoverable without duplicating volatile -details across documentation owners. - -### Required Changes - -1. Update `docs/policy/architecture.md` with the normative boundary: - deployments own cache access policy; Notarius owns confined paths, atomic - publication, validation, and recovery. Do not generalize the rule to output - or debug state in this work. -2. Update `docs/operations.md`, the canonical operational owner, to: - - replace claims that cache entries are forcibly `0700`/`0600`; - - explain requested maximum modes `0777`/`0666`, umask filtering, default - ACLs, and setgid group inheritance without claiming literal mode-bit - inheritance; - - provide concise private (`0077`) and trusted-group (`0007` plus a - deployment-provisioned setgid root, or an equivalent default ACL) - examples; - - state that cache data is reconstructible but may contain source-derived - content and operators own access and retention; - - explain that pre-feature cache trees retain their existing inode modes and - must be deliberately recreated or administratively migrated; and - - accurately limit concurrent-use claims to the existing atomicity and - validation guarantees. -3. Update `docs/config.md` only to clarify that selecting chunk-plan or - checkpoint directories does not configure their filesystem access policy. - Do not document a nonexistent permission field. -4. Update `docs/internal/state.md` with the two implemented writer paths: - rooted chunk-plan atomic creation and checkpoint use of the - process-permission file writer. Link to Operations for operator-facing mode - and provisioning guidance instead of duplicating it. -5. Review `docs/internal/overview.md` and `docs/development.md`; update links or - routing only if the existing navigation no longer reaches the canonical - state and operations documents. -6. Do not edit a historical release note. The release procedure will summarize - this change when the next release candidate is prepared. - -### Validation - -- Verify every changed relative Markdown link and heading anchor. -- Search current-behavior documentation for stale assertions that cache modes - are always `0700`/`0600`; distinguish cache from still-valid debug claims. -- Confirm current-behavior documentation outside `docs/roadmap/` contains - neither `cache.access` nor instructions for Notarius-managed - chmod/chown/chgrp/ACL mutation. -- Run `git diff --check`. - -### Stage Acceptance - -Each policy and operational fact has one canonical owner, current docs describe -only implemented behavior, and the feature roadmap remains the owner of future -status until final review. This stage is small enough for one implementation -prompt. - -## Stage 5: Repository-Wide Verification And Contract Audit - -**Completed:** 2026-09-11 - -### Goal - -Prove the complete target state and catch accidental permission, safety, or -documentation drift before review. - -### Required Review - -1. Inspect production cache code for remaining `0700`, `0600`, or `Chmod` - operations. Every remaining occurrence must be a fixture, an unrelated - output/debug contract, or otherwise justified; neither cache writer may - enforce those values. -2. Search configuration, CLI, examples, and documentation to confirm no - `cache.access` surface or private/group application policy was introduced. -3. Confirm output and debug production paths and their contract tests are - unchanged except for necessary documentation disambiguation. -4. Confirm cache reads remain side-effect free where previously guaranteed and - every cache write remains confined and atomic. -5. Review all new tests against `docs/policy/testing.md`. Consolidate redundant - mode assertions while retaining one owner for each realistic regression: - kernel filtering, existing-root preservation, each cache writer, and - assembled CLI wiring. -6. Review the ADR and canonical docs for the agreed scope. Do not describe - Unix permissions as literal inheritance from a parent directory. - -### Validation - -Run the complete offline suite: - -```sh -gofmt -w internal/core/fileio/*.go internal/framework/chunkplan/*.go internal/framework/checkpoint/*.go internal/cli/*.go go test ./... -go test -race ./internal/core/fileio ./internal/framework/chunkplan ./internal/framework/checkpoint ./internal/cli go vet ./... go build ./cmd/notarius go run ./cmd/notarius config validate --config examples/dnd-minimal.config.yml --pipeline dnd-session -OPENROUTER_API_KEY=validation-placeholder go run ./cmd/notarius config validate --config examples/dnd-complete.config.yml --pipeline dnd-session +go run ./cmd/notarius config validate --config examples/dnd-complete.config.yml --pipeline dnd-session +go run ./cmd/notarius config validate --config examples/dnd-complete.config.yml --config-profile development --pipeline dnd-session +go run ./cmd/notarius pipelines list --config examples/dnd-complete.config.yml --config-profile development --json +go run ./cmd/notarius config effective --config examples/dnd-complete.config.yml --config-profile development +go run ./cmd/notarius config explain --config examples/dnd-complete.config.yml --config-profile development /concurrency/total_llm git diff --check ``` -The config validations must perform no provider requests. Also inspect -`git status --short --untracked-files=all` for unintended files, generated -artifacts, or unrelated edits. +Use the project's normal link-check mechanism if present; otherwise check +changed relative links and heading anchors manually. Live model evaluation is +unnecessary for a configuration migration. -### Final Acceptance +**Prompt size:** Medium; verification and focused documentation reconciliation, +with no planned production redesign. -Every acceptance criterion in the feature roadmap is supported by direct code, -test, or documentation evidence; all prescribed commands pass; the working -diff contains only this feature; and the two roadmap documents remain in place -for the post-implementation review. This stage is small enough for one final -verification prompt. +## Upstream Prerequisite + +### Source Origins During ConfigKit Resolution + +**Evidence:** In v0.2.0, `Resolution` exports `IsExplicit`, `Record`, `Error`, +and `Warning`, with generic `Default` and `Normalize` helpers. It does not +expose source metadata. `Result.Metadata()` and `Result.Sources()` become +available only after `Load` returns. Mutating `Result.Config` afterward does +not change `WriteEffective` or `WriteExplanation`, which use frozen snapshots. +Therefore a single v0.2.0 load cannot both use declaration origins to normalize +paths and record that normalization in ConfigKit's inspection history. + +**Selected approach:** Request a small upstream API exposing surviving +input origins and canonical source records during the existing resolution +callback. It should distinguish base/profile/CLI/default origins, identify the +declaring file for a scalar, and permit lookup at original paths before +application map-key normalization. Keep the existing recorded mutation API. +Exact method names and the containing release must be confirmed upstream; +none are assumed to exist by this plan. This preserves one load, ConfigKit +ownership of composition/provenance, and the agreed effective-output contract. +The tradeoff is an upstream release dependency at Stage 7. This plan does not +authorize changes to the sibling repository. + +**Alternative considered:** Stay on v0.2.0 and resolve paths in the narrow Notarius +runtime converter using completed-result metadata. Preserve original document +paths for origin lookup. Present ConfigKit's effective configuration as a +logical configuration view, and expose resolved runtime paths separately in +debug/inspection output. This is implementable without an upstream change and +keeps composition centralized, but changes the agreed inspection contract, +requires Notarius to own the extra path report, and cannot make ConfigKit's +explanation record the post-load normalization. It requires explicit approval +and revision of Stage 7 and the feature roadmap before dependent work begins; +the user selected the upstream extension instead. + +Do not resolve this by silently dropping imports/profiles, changing all file +paths to working-directory-relative, reparsing YAML, or performing a second +load that might observe different files. All other product decisions in this +plan are settled; the upstream release is the sole remaining gate. No product +questions remain open. Once the API is released, review its exact signatures, +update the pinned version and this stage's API mapping, and then resume Stage 7.