Add initial documentation skeleton

2026-09-06 10:52:38 -05:00
parent a10bb53628
commit 18524330fc
8 changed files with 1663 additions and 0 deletions
--- a/CONFIGKIT_DESIGN_PROPOSAL.md
+++ b/CONFIGKIT_DESIGN_PROPOSAL.md
@@ -0,0 +1,861 @@
+# ConfigKit — Initial Design and Architecture
+
+## Status
+
+**Design note for initial implementation.**
+
+Audience: future maintainers of ConfigKit, including LLM coding agents assisting with implementation.
+
+This document captures the current architectural and policy decisions for ConfigKit. The goal is to establish a sufficiently opinionated v1 design that can be implemented consistently across the application suite without prematurely becoming a general-purpose configuration framework.
+
+---
+
+## 1. Purpose
+
+ConfigKit is a shared Go library for configuration loading and resolution across applications such as Seriatim, Audita, Narratio, WeatherFeeder, WeatherReporter, Distributor, and related tools.
+
+Its purpose is to provide a consistent, blessed configuration lifecycle for applications that use YAML configuration files.
+
+ConfigKit should solve recurring infrastructure concerns once:
+
+- file loading;
+- YAML parsing and strictness;
+- deterministic imports / `conf.d` composition;
+- duplicate-definition detection;
+- intentional higher-precedence overlays such as profiles;
+- presence tracking;
+- source/provenance tracking;
+- typed decoding;
+- standardized diagnostics;
+- effective-configuration inspection;
+- support for application defaults, derived defaults, and validation.
+
+ConfigKit is **not** intended to become a universal configuration framework.
+
+---
+
+## 2. Design Principles
+
+### 2.1 YAML is syntax; ConfigKit defines semantics
+
+The YAML parser must not implicitly define ConfigKit behavior for:
+
+- composition;
+- precedence;
+- duplicate keys;
+- sequence merging;
+- `null`;
+- imports;
+- defaults;
+- profile overlays.
+
+These behaviors must be explicit ConfigKit policy.
+
+### 2.2 Preserve information until late in the pipeline
+
+ConfigKit should retain the parsed YAML tree and source metadata through composition and overlay processing.
+
+Do **not** immediately reduce configuration to `map[string]any` or typed Go structs.
+
+The implementation should preserve enough information to answer:
+
+> Why does this effective value have this value, and where did it come from?
+
+### 2.3 Keep application-facing Go types ordinary
+
+Applications should normally use idiomatic types:
+
+```go
+type Config struct {
+    Enabled bool
+    Workers int
+    Timeout time.Duration
+}
+```
+
+ConfigKit should not require pervasive `*bool`, `*int`, or `Optional[T]` fields merely to distinguish absence from explicit zero values.
+
+Parser-level presence is metadata owned by ConfigKit.
+
+Application-level optionality should remain explicit in the application's Go type where it is genuinely part of the application's semantics.
+
+---
+
+## 3. Responsibility Boundary
+
+### ConfigKit owns
+
+- configuration source loading;
+- YAML parsing;
+- YAML policy enforcement;
+- import discovery;
+- deterministic import ordering;
+- strict composition;
+- overlay application;
+- duplicate-definition detection;
+- source location tracking;
+- presence tracking;
+- definition history / provenance;
+- typed decoding;
+- standard diagnostics;
+- effective-config rendering;
+- config source reporting;
+- config fingerprinting, if implemented;
+- redaction mechanisms for secrets.
+
+### Consuming applications own
+
+- their `Config` Go types;
+- static default values;
+- derived/defaulted values;
+- semantic validation;
+- schema meaning;
+- schema migrations;
+- runtime initialization;
+- runtime resource validation;
+- hot reload and resource replacement.
+
+---
+
+## 4. Configuration Resolution Model
+
+The conceptual lifecycle is:
+
+```text
+source discovery
+      ↓
+YAML parsing
+      ↓
+YAML policy validation
+      ↓
+strict base composition
+      ↓
+profile overlay
+      ↓
+static defaults / typed decode
+      ↓
+derived defaults / normalization
+      ↓
+semantic validation
+      ↓
+immutable effective configuration
+```
+
+A useful precedence model is:
+
+```text
+LOW PRECEDENCE
+
+application defaults
+        ↓
+base configuration
+  ├─ main file
+  └─ imports / conf.d     ← strict composition
+        ↓
+selected profile         ← intentional overlay
+
+HIGH PRECEDENCE
+```
+
+Future layers such as environment-variable or CLI value overrides may be added later, but are not required for v1.
+
+---
+
+## 5. YAML Policy
+
+YAML remains the configuration format.
+
+ConfigKit should support a conservative YAML subset rather than arbitrary YAML semantics.
+
+Recommended v1 policy:
+
+- one YAML document per file;
+- mappings, sequences, and scalars;
+- string mapping keys;
+- comments;
+- unique mapping keys;
+- reject unknown fields when decoding into typed configuration;
+- reject custom YAML tags;
+- reject YAML merge keys (`<<`);
+- strongly consider rejecting anchors / aliases;
+- reject ambiguous or unsupported constructs rather than guessing.
+
+The intended model is approximately:
+
+> JSON's data model, with comments and more pleasant human syntax.
+
+The underlying YAML implementation must remain an internal implementation detail. Public ConfigKit APIs should not expose parser-specific node or error types.
+
+---
+
+## 6. Presence Semantics
+
+ConfigKit must distinguish at least these states:
+
+| State | Meaning |
+|---|---|
+| key absent | no opinion; a lower-precedence/default value may apply |
+| `key: false`, `key: 0`, `key: ""` | explicitly configured value |
+| `key: null` | explicitly null; not equivalent to absent |
+| ordinary nonzero value | explicitly configured value |
+
+Presence information must survive configuration resolution even after the final typed Go value is produced.
+
+Applications should be able to determine, when necessary, whether an effective value was explicitly configured or defaulted.
+
+Conceptual metadata API:
+
+```go
+result.Config
+result.Metadata.IsExplicit("transcription.concurrency")
+result.Metadata.Origin("transcription.concurrency")
+result.Metadata.History("transcription.concurrency")
+```
+
+Exact API names are not yet fixed.
+
+---
+
+## 7. Defaults
+
+### 7.1 Static defaults
+
+Applications define static defaults, for example:
+
+```go
+func DefaultConfig() Config {
+    return Config{
+        Enabled: true,
+        Workers: 4,
+        Timeout: 30 * time.Second,
+    }
+}
+```
+
+Explicit YAML values such as `false` and `0` must override those defaults.
+
+Absent YAML values leave defaults intact.
+
+### 7.2 Derived defaults
+
+Defaults that depend on other resolved fields should be applied by an application-owned resolver after typed decoding.
+
+Example:
+
+```go
+func ResolveConfig(cfg *Config, meta configkit.Metadata) error {
+    if !meta.IsExplicit("timeout") {
+        cfg.Timeout = cfg.PollInterval / 10
+    }
+    return nil
+}
+```
+
+ConfigKit should provide a standard lifecycle hook for this but should not own application-specific derived-default logic.
+
+---
+
+## 8. `null`
+
+`null` is **not** equivalent to absence.
+
+Recommended v1 rule:
+
+> `null` is an explicit value and is valid only where the application schema permits null.
+
+For ordinary scalar fields such as:
+
+```go
+Workers int
+Enabled bool
+```
+
+this should be invalid:
+
+```yaml
+workers: null
+```
+
+Nullable application semantics may be represented using suitable Go types such as pointers.
+
+Do not use `null` in v1 to mean "reset this value to the application default."
+
+If profile-level reset-to-default semantics are eventually required, add an explicit ConfigKit mechanism rather than overloading YAML `null`.
+
+---
+
+## 9. Strict Composition
+
+Base configuration may be assembled from a main file plus imported fragments, including Debian-style `conf.d` directories.
+
+Composition means:
+
+> Multiple files contribute non-conflicting portions of one logical configuration.
+
+A duplicate semantic value path across composed files is an error.
+
+Example allowed:
+
+```yaml
+# 10-server.yaml
+server:
+  host: localhost
+```
+
+```yaml
+# 20-server.yaml
+server:
+  port: 8080
+```
+
+These define different paths:
+
+```text
+server.host
+server.port
+```
+
+Example rejected:
+
+```yaml
+# A
+server:
+  port: 8080
+```
+
+```yaml
+# B
+server:
+  port: 9000
+```
+
+Both define:
+
+```text
+server.port
+```
+
+The diagnostic should identify both source locations.
+
+### 9.1 Mapping behavior
+
+Mappings compose recursively by semantic path.
+
+### 9.2 Sequence behavior
+
+Sequences do not implicitly compose or merge.
+
+If two composed files define the same sequence path, that is a collision.
+
+When configuration needs independently composable named entries, prefer mappings keyed by stable identifiers rather than lists.
+
+---
+
+## 10. Intentional Overlay
+
+Overlay is a separate operation from composition.
+
+Overlay means:
+
+> A higher-precedence layer intentionally replaces values from a lower-precedence layer.
+
+For mappings, overlay recursively by key.
+
+For scalar values, the higher-precedence value wins.
+
+For sequences, the higher-precedence sequence replaces the lower-precedence sequence in full.
+
+Do not implement implicit sequence append, union, or index-wise merging in v1.
+
+---
+
+## 11. Profiles
+
+Profiles are the first intended overlay mechanism.
+
+Example layout:
+
+```text
+config/
+├── narratio.yml
+├── conf.d/
+│   ├── 10-transcription.yml
+│   ├── 20-artifacts.yml
+│   └── 30-distribution.yml
+└── profiles/
+    ├── production.yml
+    └── testing.yml
+```
+
+Default invocation loads only the composed base configuration.
+
+Example:
+
+```text
+narratio
+```
+
+Explicit profile selection:
+
+```text
+narratio --config-profile=testing
+```
+
+produces:
+
+```text
+application defaults
+        ↓
+main config + imports
+        ↓
+profiles/testing.yml
+```
+
+### 11.1 `default` profile
+
+"Default" should mean **no profile overlay**.
+
+Do not require a literal `profiles/default.yml`.
+
+### 11.2 Profile names
+
+`--config-profile` should accept a profile name, not an arbitrary path.
+
+Profile names should use a constrained character set, e.g.:
+
+```text
+[a-zA-Z0-9_-]+
+```
+
+The selected profile should resolve deterministically relative to the configuration root.
+
+### 11.3 Profiles should not redefine the configuration graph
+
+Profiles should modify application configuration values, not change how ConfigKit discovers configuration.
+
+Profile overlays should therefore not introduce new imports or redefine import behavior.
+
+The base configuration graph is discovered and composed first; the selected profile is then applied to the resulting configuration tree.
+
+---
+
+## 12. Import Policy
+
+Import behavior must be deterministic and explicitly specified.
+
+Recommended v1 rules:
+
+- relative imports resolve relative to the file containing the import;
+- glob results are sorted deterministically;
+- duplicate semantic values across imported fragments are errors;
+- explicit missing imports are errors;
+- an optional empty `conf.d` glob may resolve to zero files;
+- import cycles are errors;
+- cycle diagnostics should show the full cycle;
+- define file identity carefully with respect to symlinks;
+- consider constraining imports to an allowed configuration root;
+- do not support remote/network imports in v1.
+
+Remote imports create separate concerns around availability, caching, authentication, integrity, startup latency, and freshness and should remain out of scope.
+
+---
+
+## 13. Provenance and Definition History
+
+Provenance is a first-class design requirement.
+
+ConfigKit should retain, for each effective semantic value path:
+
+- whether it was absent or explicitly configured;
+- its effective value source;
+- source file;
+- line and column where practical;
+- configuration layer;
+- previous definitions overridden by higher-precedence layers.
+
+Conceptually:
+
+```go
+type Definition struct {
+    Layer  Layer
+    Source Source
+    Value  NodeRef
+}
+
+type FieldHistory struct {
+    Definitions []Definition
+}
+```
+
+Possible layers:
+
+```go
+const (
+    LayerDefault Layer = iota
+    LayerBase
+    LayerProfile
+)
+```
+
+Future layers may include environment or CLI overrides.
+
+This metadata should enable diagnostics such as:
+
+```text
+transcription.concurrency = 1
+
+history:
+  application default                     4
+  conf.d/10-transcription.yml:14          8
+  profiles/testing.yml:7                  1
+
+effective source:
+  profiles/testing.yml:7
+```
+
+---
+
+## 14. Typed Decoding
+
+After composition and overlays produce one effective presence-aware configuration tree, ConfigKit decodes it into the application's typed Go `Config`.
+
+Requirements:
+
+- unknown fields are errors;
+- type mismatches are errors;
+- diagnostics should preserve source location where possible;
+- parser-specific errors should be converted into ConfigKit diagnostics.
+
+Do not expose raw YAML parser types through the public API.
+
+---
+
+## 15. Semantic Validation
+
+Application-specific validation remains application-owned.
+
+ConfigKit should provide a standard validation hook and diagnostic type.
+
+Validation should normally collect multiple independent errors rather than aborting after the first.
+
+Example output:
+
+```text
+configuration invalid:
+
+  weather.sources.nws.timeout
+    must be > 0
+    config.yml:19:5
+
+  outputs.public
+    duplicate definition
+    config.yml:42:3
+    conf.d/90-output.yml:8:3
+```
+
+---
+
+## 16. Schema Versioning
+
+Applications should support explicit configuration schema versions.
+
+Example:
+
+```yaml
+schema_version: narratio.v1
+```
+
+The version identifies the application's configuration schema, not the ConfigKit library version.
+
+ConfigKit may later provide lifecycle support for application-provided migrations, but automatic migration is not required for the initial implementation.
+
+The design should leave a clean pipeline location for schema-version checking and migration.
+
+---
+
+## 17. Secrets and Redaction
+
+ConfigKit will eventually process configuration containing secrets.
+
+Redaction must be considered before implementing:
+
+- effective-config output;
+- diagnostics;
+- config diffs;
+- logging;
+- config fingerprinting.
+
+Potential application annotation:
+
+```go
+APIKey string `yaml:"api_key" configkit:"secret"`
+```
+
+Exact mechanism is TBD.
+
+Human-facing tooling must never accidentally emit raw secrets.
+
+If fingerprints include secret values, the raw values must still never be exposed.
+
+---
+
+## 18. Diagnostics and Operability
+
+ConfigKit should establish recognizable diagnostics across the application suite.
+
+Potential standardized commands enabled by ConfigKit:
+
+```text
+<app> config validate
+<app> config effective
+<app> config sources
+<app> config explain <path>
+```
+
+These commands do not all need to exist in the initial skeleton, but the metadata model should make them straightforward to implement later.
+
+`config effective` must support secret redaction.
+
+---
+
+## 19. Non-Goals for v1
+
+Do not include these in the initial implementation unless a concrete application requirement appears:
+
+- hot reload;
+- runtime resource replacement;
+- remote/network imports;
+- arbitrary configuration providers;
+- TOML/JSON/HCL/CUE support;
+- Vault/S3 configuration providers;
+- sophisticated environment-variable overlays;
+- arbitrary CLI field overrides;
+- sequence append/union merge strategies;
+- arbitrary custom merge policies;
+- profile inheritance;
+- reset-to-default operators;
+- application runtime validation such as checking whether an S3 bucket exists.
+
+ConfigKit should remain opinionated.
+
+---
+
+## 20. Initial Internal Architecture
+
+A reasonable internal model is:
+
+```text
+Source
+  ↓
+Document
+  ├─ parsed tree
+  └─ source locations
+  ↓
+Composer
+  strict combination
+  duplicate definitions = error
+  ↓
+Overlay
+  higher layer wins
+  mappings recurse
+  sequences replace
+  ↓
+Metadata / Provenance
+  presence
+  origins
+  definition history
+  ↓
+Decoder[T]
+  effective tree → typed Config
+  ↓
+Resolver[T]
+  application-derived defaults / normalization
+  ↓
+Validator[T]
+  application semantic validation
+  ↓
+Result[T]
+```
+
+Conceptual result:
+
+```go
+type Result[T any] struct {
+    Config      T
+    Metadata    Metadata
+    Sources     []Source
+    Diagnostics []Diagnostic
+    Fingerprint string // optional / future
+}
+```
+
+---
+
+## 21. Possible Package Skeleton
+
+Exact names are not settled, but an initial repository could look roughly like:
+
+```text
+configkit/
+├── configkit.go
+├── source.go
+├── document.go
+├── compose.go
+├── overlay.go
+├── metadata.go
+├── decode.go
+├── diagnostic.go
+├── profile.go
+├── import.go
+├── schema.go
+├── secret.go
+└── internal/
+    └── yaml/
+```
+
+Public API should remain small.
+
+A future happy path might resemble:
+
+```go
+result, err := configkit.Load[Config](
+    configkit.File(path),
+    configkit.WithImports(),
+    configkit.WithProfile(profile),
+    configkit.WithDefaults(DefaultConfig()),
+    configkit.WithResolver(ResolveConfig),
+    configkit.WithValidator(ValidateConfig),
+)
+```
+
+This is illustrative only. Do not prematurely lock in option names or signatures.
+
+---
+
+## 22. Suggested Initial Implementation Sequence
+
+### Stage 1 — Core document model
+
+Implement:
+
+- local file source;
+- YAML parsing;
+- conservative YAML validation;
+- source locations;
+- semantic path traversal;
+- internal presence-aware tree representation.
+
+### Stage 2 — Strict composition
+
+Implement:
+
+- recursive mapping composition;
+- duplicate semantic-path detection;
+- deterministic import ordering;
+- useful duplicate diagnostics.
+
+### Stage 3 — Profile overlay
+
+Implement:
+
+- constrained profile name resolution;
+- recursive mapping overlay;
+- scalar replacement;
+- sequence replacement;
+- provenance history across layers.
+
+### Stage 4 — Typed decoding
+
+Implement:
+
+- generic `Load[T]` or equivalent;
+- strict unknown-field rejection;
+- typed errors converted to ConfigKit diagnostics.
+
+### Stage 5 — Defaults and resolution hooks
+
+Implement:
+
+- static application defaults;
+- explicit-value preservation for `false`, `0`, and empty strings;
+- metadata query for explicitness;
+- derived-default / resolver hook.
+
+### Stage 6 — Semantic validation
+
+Implement:
+
+- application validation hook;
+- multi-error diagnostics.
+
+### Stage 7 — Introspection
+
+Implement or prototype:
+
+- source listing;
+- effective-config rendering;
+- `explain(path)`;
+- secret redaction.
+
+---
+
+## 23. Tests That Define Policy
+
+ConfigKit should have policy-focused tests, not merely implementation tests.
+
+At minimum, test:
+
+- absent bool receives default;
+- explicit `false` overrides `true` default;
+- absent integer receives default;
+- explicit `0` overrides nonzero default;
+- explicit empty string remains explicit;
+- `null` differs from absent;
+- duplicate key within one YAML file fails;
+- duplicate semantic path across composed files fails;
+- sibling mapping paths compose successfully;
+- profile scalar overrides base scalar;
+- profile mapping recursively overrides only specified children;
+- profile sequence replaces base sequence;
+- base sequence collision across composed files fails;
+- profile cannot redefine import graph;
+- imports are deterministically ordered;
+- import cycles fail with useful diagnostics;
+- unknown typed-config fields fail;
+- provenance identifies effective source;
+- provenance retains overridden definitions;
+- defaults are distinguishable from explicit equal-valued configuration;
+- secrets are redacted from introspection output.
+
+These tests should become executable documentation for ConfigKit policy.
+
+---
+
+## 24. Key Decisions Summary
+
+For v1:
+
+1. YAML remains the configuration language.
+2. ConfigKit defines a conservative YAML policy.
+3. Parsed configuration remains presence-aware until after composition and overlays.
+4. Application-facing Go structs remain idiomatic and strongly typed.
+5. Absent, explicit zero/false/empty, and `null` are distinct.
+6. Base imports use strict composition.
+7. Duplicate semantic values across composed files are errors.
+8. Profiles use explicit higher-precedence overlay semantics.
+9. Mapping overlays recurse.
+10. Sequence overlays replace.
+11. "Default profile" means no profile overlay.
+12. Profiles are names, not arbitrary file paths.
+13. Profiles do not redefine the import graph.
+14. Static defaults are application-owned.
+15. Derived defaults are application-owned and presence-aware.
+16. Provenance and definition history are first-class metadata.
+17. Semantic validation is application-owned but follows a common lifecycle.
+18. Parser implementation details remain private.
+19. Secrets must be redactable.
+20. Hot reload, remote providers, generalized merge strategies, and broad extensibility are non-goals until concrete requirements exist.
+
+The core architectural objective is:
+
+> **Build one deterministic, explainable configuration resolution model for the entire application suite.**