eric/ConfigKit

Fork 0

Files

Eric Rakestraw 18524330fc Add initial documentation skeleton

2026-09-06 10:52:38 -05:00

20 KiB

Raw Blame History

ConfigKit — Initial Design and Architecture

Status

Design note for initial implementation.

Audience: future maintainers of ConfigKit, including LLM coding agents assisting with implementation.

This document captures the current architectural and policy decisions for ConfigKit. The goal is to establish a sufficiently opinionated v1 design that can be implemented consistently across the application suite without prematurely becoming a general-purpose configuration framework.

1. Purpose

ConfigKit is a shared Go library for configuration loading and resolution across applications such as Seriatim, Audita, Narratio, WeatherFeeder, WeatherReporter, Distributor, and related tools.

Its purpose is to provide a consistent, blessed configuration lifecycle for applications that use YAML configuration files.

ConfigKit should solve recurring infrastructure concerns once:

file loading;
YAML parsing and strictness;
deterministic imports / conf.d composition;
duplicate-definition detection;
intentional higher-precedence overlays such as profiles;
presence tracking;
source/provenance tracking;
typed decoding;
standardized diagnostics;
effective-configuration inspection;
support for application defaults, derived defaults, and validation.

ConfigKit is not intended to become a universal configuration framework.

2. Design Principles

2.1 YAML is syntax; ConfigKit defines semantics

The YAML parser must not implicitly define ConfigKit behavior for:

composition;
precedence;
duplicate keys;
sequence merging;
null;
imports;
defaults;
profile overlays.

These behaviors must be explicit ConfigKit policy.

2.2 Preserve information until late in the pipeline

ConfigKit should retain the parsed YAML tree and source metadata through composition and overlay processing.

Do not immediately reduce configuration to map[string]any or typed Go structs.

The implementation should preserve enough information to answer:

Why does this effective value have this value, and where did it come from?

2.3 Keep application-facing Go types ordinary

Applications should normally use idiomatic types:

type Config struct {
    Enabled bool
    Workers int
    Timeout time.Duration
}

ConfigKit should not require pervasive *bool, *int, or Optional[T] fields merely to distinguish absence from explicit zero values.

Parser-level presence is metadata owned by ConfigKit.

Application-level optionality should remain explicit in the application's Go type where it is genuinely part of the application's semantics.

3. Responsibility Boundary

ConfigKit owns

configuration source loading;
YAML parsing;
YAML policy enforcement;
import discovery;
deterministic import ordering;
strict composition;
overlay application;
duplicate-definition detection;
source location tracking;
presence tracking;
definition history / provenance;
typed decoding;
standard diagnostics;
effective-config rendering;
config source reporting;
config fingerprinting, if implemented;
redaction mechanisms for secrets.

Consuming applications own

their Config Go types;
static default values;
derived/defaulted values;
semantic validation;
schema meaning;
schema migrations;
runtime initialization;
runtime resource validation;
hot reload and resource replacement.

4. Configuration Resolution Model

The conceptual lifecycle is:

source discovery
      ↓
YAML parsing
      ↓
YAML policy validation
      ↓
strict base composition
      ↓
profile overlay
      ↓
static defaults / typed decode
      ↓
derived defaults / normalization
      ↓
semantic validation
      ↓
immutable effective configuration

A useful precedence model is:

LOW PRECEDENCE

application defaults
        ↓
base configuration
  ├─ main file
  └─ imports / conf.d     ← strict composition
        ↓
selected profile         ← intentional overlay

HIGH PRECEDENCE

Future layers such as environment-variable or CLI value overrides may be added later, but are not required for v1.

5. YAML Policy

YAML remains the configuration format.

ConfigKit should support a conservative YAML subset rather than arbitrary YAML semantics.

Recommended v1 policy:

one YAML document per file;
mappings, sequences, and scalars;
string mapping keys;
comments;
unique mapping keys;
reject unknown fields when decoding into typed configuration;
reject custom YAML tags;
reject YAML merge keys (<<);
strongly consider rejecting anchors / aliases;
reject ambiguous or unsupported constructs rather than guessing.

The intended model is approximately:

JSON's data model, with comments and more pleasant human syntax.

The underlying YAML implementation must remain an internal implementation detail. Public ConfigKit APIs should not expose parser-specific node or error types.

6. Presence Semantics

ConfigKit must distinguish at least these states:

State	Meaning
key absent	no opinion; a lower-precedence/default value may apply
`key: false`, `key: 0`, `key: ""`	explicitly configured value
`key: null`	explicitly null; not equivalent to absent
ordinary nonzero value	explicitly configured value

Presence information must survive configuration resolution even after the final typed Go value is produced.

Applications should be able to determine, when necessary, whether an effective value was explicitly configured or defaulted.

Conceptual metadata API:

result.Config
result.Metadata.IsExplicit("transcription.concurrency")
result.Metadata.Origin("transcription.concurrency")
result.Metadata.History("transcription.concurrency")

Exact API names are not yet fixed.

7. Defaults

7.1 Static defaults

Applications define static defaults, for example:

func DefaultConfig() Config {
    return Config{
        Enabled: true,
        Workers: 4,
        Timeout: 30 * time.Second,
    }
}

Explicit YAML values such as false and 0 must override those defaults.

Absent YAML values leave defaults intact.

7.2 Derived defaults

Defaults that depend on other resolved fields should be applied by an application-owned resolver after typed decoding.

Example:

func ResolveConfig(cfg *Config, meta configkit.Metadata) error {
    if !meta.IsExplicit("timeout") {
        cfg.Timeout = cfg.PollInterval / 10
    }
    return nil
}

ConfigKit should provide a standard lifecycle hook for this but should not own application-specific derived-default logic.

8. `null`

null is not equivalent to absence.

Recommended v1 rule:

null is an explicit value and is valid only where the application schema permits null.

For ordinary scalar fields such as:

Workers int
Enabled bool

this should be invalid:

workers: null

Nullable application semantics may be represented using suitable Go types such as pointers.

Do not use null in v1 to mean "reset this value to the application default."

If profile-level reset-to-default semantics are eventually required, add an explicit ConfigKit mechanism rather than overloading YAML null.

9. Strict Composition

Base configuration may be assembled from a main file plus imported fragments, including Debian-style conf.d directories.

Composition means:

Multiple files contribute non-conflicting portions of one logical configuration.

A duplicate semantic value path across composed files is an error.

Example allowed:

# 10-server.yaml
server:
  host: localhost

# 20-server.yaml
server:
  port: 8080

These define different paths:

server.host
server.port

Example rejected:

# A
server:
  port: 8080

# B
server:
  port: 9000

Both define:

server.port

The diagnostic should identify both source locations.

9.1 Mapping behavior

Mappings compose recursively by semantic path.

9.2 Sequence behavior

Sequences do not implicitly compose or merge.

If two composed files define the same sequence path, that is a collision.

When configuration needs independently composable named entries, prefer mappings keyed by stable identifiers rather than lists.

10. Intentional Overlay

Overlay is a separate operation from composition.

Overlay means:

A higher-precedence layer intentionally replaces values from a lower-precedence layer.

For mappings, overlay recursively by key.

For scalar values, the higher-precedence value wins.

For sequences, the higher-precedence sequence replaces the lower-precedence sequence in full.

Do not implement implicit sequence append, union, or index-wise merging in v1.

11. Profiles

Profiles are the first intended overlay mechanism.

Example layout:

config/
├── narratio.yml
├── conf.d/
│   ├── 10-transcription.yml
│   ├── 20-artifacts.yml
│   └── 30-distribution.yml
└── profiles/
    ├── production.yml
    └── testing.yml

Default invocation loads only the composed base configuration.

Example:

narratio

Explicit profile selection:

narratio --config-profile=testing

produces:

application defaults
        ↓
main config + imports
        ↓
profiles/testing.yml

11.1 `default` profile

"Default" should mean no profile overlay.

Do not require a literal profiles/default.yml.

11.2 Profile names

--config-profile should accept a profile name, not an arbitrary path.

Profile names should use a constrained character set, e.g.:

[a-zA-Z0-9_-]+

The selected profile should resolve deterministically relative to the configuration root.

11.3 Profiles should not redefine the configuration graph

Profiles should modify application configuration values, not change how ConfigKit discovers configuration.

Profile overlays should therefore not introduce new imports or redefine import behavior.

The base configuration graph is discovered and composed first; the selected profile is then applied to the resulting configuration tree.

12. Import Policy

Import behavior must be deterministic and explicitly specified.

Recommended v1 rules:

relative imports resolve relative to the file containing the import;
glob results are sorted deterministically;
duplicate semantic values across imported fragments are errors;
explicit missing imports are errors;
an optional empty conf.d glob may resolve to zero files;
import cycles are errors;
cycle diagnostics should show the full cycle;
define file identity carefully with respect to symlinks;
consider constraining imports to an allowed configuration root;
do not support remote/network imports in v1.

Remote imports create separate concerns around availability, caching, authentication, integrity, startup latency, and freshness and should remain out of scope.

13. Provenance and Definition History

Provenance is a first-class design requirement.

ConfigKit should retain, for each effective semantic value path:

whether it was absent or explicitly configured;
its effective value source;
source file;
line and column where practical;
configuration layer;
previous definitions overridden by higher-precedence layers.

Conceptually:

type Definition struct {
    Layer  Layer
    Source Source
    Value  NodeRef
}

type FieldHistory struct {
    Definitions []Definition
}

Possible layers:

const (
    LayerDefault Layer = iota
    LayerBase
    LayerProfile
)

Future layers may include environment or CLI overrides.

This metadata should enable diagnostics such as:

transcription.concurrency = 1

history:
  application default                     4
  conf.d/10-transcription.yml:14          8
  profiles/testing.yml:7                  1

effective source:
  profiles/testing.yml:7

14. Typed Decoding

After composition and overlays produce one effective presence-aware configuration tree, ConfigKit decodes it into the application's typed Go Config.

Requirements:

unknown fields are errors;
type mismatches are errors;
diagnostics should preserve source location where possible;
parser-specific errors should be converted into ConfigKit diagnostics.

Do not expose raw YAML parser types through the public API.

15. Semantic Validation

Application-specific validation remains application-owned.

ConfigKit should provide a standard validation hook and diagnostic type.

Validation should normally collect multiple independent errors rather than aborting after the first.

Example output:

configuration invalid:

  weather.sources.nws.timeout
    must be > 0
    config.yml:19:5

  outputs.public
    duplicate definition
    config.yml:42:3
    conf.d/90-output.yml:8:3

16. Schema Versioning

Applications should support explicit configuration schema versions.

Example:

schema_version: narratio.v1

The version identifies the application's configuration schema, not the ConfigKit library version.

ConfigKit may later provide lifecycle support for application-provided migrations, but automatic migration is not required for the initial implementation.

The design should leave a clean pipeline location for schema-version checking and migration.

17. Secrets and Redaction

ConfigKit will eventually process configuration containing secrets.

Redaction must be considered before implementing:

effective-config output;
diagnostics;
config diffs;
logging;
config fingerprinting.

Potential application annotation:

APIKey string `yaml:"api_key" configkit:"secret"`

Exact mechanism is TBD.

Human-facing tooling must never accidentally emit raw secrets.

If fingerprints include secret values, the raw values must still never be exposed.

18. Diagnostics and Operability

ConfigKit should establish recognizable diagnostics across the application suite.

Potential standardized commands enabled by ConfigKit:

<app> config validate
<app> config effective
<app> config sources
<app> config explain <path>

These commands do not all need to exist in the initial skeleton, but the metadata model should make them straightforward to implement later.

config effective must support secret redaction.

19. Non-Goals for v1

Do not include these in the initial implementation unless a concrete application requirement appears:

hot reload;
runtime resource replacement;
remote/network imports;
arbitrary configuration providers;
TOML/JSON/HCL/CUE support;
Vault/S3 configuration providers;
sophisticated environment-variable overlays;
arbitrary CLI field overrides;
sequence append/union merge strategies;
arbitrary custom merge policies;
profile inheritance;
reset-to-default operators;
application runtime validation such as checking whether an S3 bucket exists.

ConfigKit should remain opinionated.

20. Initial Internal Architecture

A reasonable internal model is:

Source
  ↓
Document
  ├─ parsed tree
  └─ source locations
  ↓
Composer
  strict combination
  duplicate definitions = error
  ↓
Overlay
  higher layer wins
  mappings recurse
  sequences replace
  ↓
Metadata / Provenance
  presence
  origins
  definition history
  ↓
Decoder[T]
  effective tree → typed Config
  ↓
Resolver[T]
  application-derived defaults / normalization
  ↓
Validator[T]
  application semantic validation
  ↓
Result[T]

Conceptual result:

type Result[T any] struct {
    Config      T
    Metadata    Metadata
    Sources     []Source
    Diagnostics []Diagnostic
    Fingerprint string // optional / future
}

21. Possible Package Skeleton

Exact names are not settled, but an initial repository could look roughly like:

configkit/
├── configkit.go
├── source.go
├── document.go
├── compose.go
├── overlay.go
├── metadata.go
├── decode.go
├── diagnostic.go
├── profile.go
├── import.go
├── schema.go
├── secret.go
└── internal/
    └── yaml/

Public API should remain small.

A future happy path might resemble:

result, err := configkit.Load[Config](
    configkit.File(path),
    configkit.WithImports(),
    configkit.WithProfile(profile),
    configkit.WithDefaults(DefaultConfig()),
    configkit.WithResolver(ResolveConfig),
    configkit.WithValidator(ValidateConfig),
)

This is illustrative only. Do not prematurely lock in option names or signatures.

22. Suggested Initial Implementation Sequence

Stage 1 — Core document model

Implement:

local file source;
YAML parsing;
conservative YAML validation;
source locations;
semantic path traversal;
internal presence-aware tree representation.

Stage 2 — Strict composition

Implement:

recursive mapping composition;
duplicate semantic-path detection;
deterministic import ordering;
useful duplicate diagnostics.

Stage 3 — Profile overlay

Implement:

constrained profile name resolution;
recursive mapping overlay;
scalar replacement;
sequence replacement;
provenance history across layers.

Stage 4 — Typed decoding

Implement:

generic Load[T] or equivalent;
strict unknown-field rejection;
typed errors converted to ConfigKit diagnostics.

Stage 5 — Defaults and resolution hooks

Implement:

static application defaults;
explicit-value preservation for false, 0, and empty strings;
metadata query for explicitness;
derived-default / resolver hook.

Stage 6 — Semantic validation

Implement:

application validation hook;
multi-error diagnostics.

Stage 7 — Introspection

Implement or prototype:

source listing;
effective-config rendering;
explain(path);
secret redaction.

23. Tests That Define Policy

ConfigKit should have policy-focused tests, not merely implementation tests.

At minimum, test:

absent bool receives default;
explicit false overrides true default;
absent integer receives default;
explicit 0 overrides nonzero default;
explicit empty string remains explicit;
null differs from absent;
duplicate key within one YAML file fails;
duplicate semantic path across composed files fails;
sibling mapping paths compose successfully;
profile scalar overrides base scalar;
profile mapping recursively overrides only specified children;
profile sequence replaces base sequence;
base sequence collision across composed files fails;
profile cannot redefine import graph;
imports are deterministically ordered;
import cycles fail with useful diagnostics;
unknown typed-config fields fail;
provenance identifies effective source;
provenance retains overridden definitions;
defaults are distinguishable from explicit equal-valued configuration;
secrets are redacted from introspection output.

These tests should become executable documentation for ConfigKit policy.

24. Key Decisions Summary

For v1:

YAML remains the configuration language.
ConfigKit defines a conservative YAML policy.
Parsed configuration remains presence-aware until after composition and overlays.
Application-facing Go structs remain idiomatic and strongly typed.
Absent, explicit zero/false/empty, and null are distinct.
Base imports use strict composition.
Duplicate semantic values across composed files are errors.
Profiles use explicit higher-precedence overlay semantics.
Mapping overlays recurse.
Sequence overlays replace.
"Default profile" means no profile overlay.
Profiles are names, not arbitrary file paths.
Profiles do not redefine the import graph.
Static defaults are application-owned.
Derived defaults are application-owned and presence-aware.
Provenance and definition history are first-class metadata.
Semantic validation is application-owned but follows a common lifecycle.
Parser implementation details remain private.
Secrets must be redactable.
Hot reload, remote providers, generalized merge strategies, and broad extensibility are non-goals until concrete requirements exist.

The core architectural objective is:

Build one deterministic, explainable configuration resolution model for the entire application suite.

20 KiB Raw Blame History

ConfigKit — Initial Design and Architecture

Status

1. Purpose

2. Design Principles

2.1 YAML is syntax; ConfigKit defines semantics

2.2 Preserve information until late in the pipeline

2.3 Keep application-facing Go types ordinary

3. Responsibility Boundary

ConfigKit owns

Consuming applications own

4. Configuration Resolution Model

5. YAML Policy

6. Presence Semantics

7. Defaults

7.1 Static defaults

7.2 Derived defaults

8. null

9. Strict Composition

9.1 Mapping behavior

9.2 Sequence behavior

10. Intentional Overlay

11. Profiles

11.1 default profile

11.2 Profile names

11.3 Profiles should not redefine the configuration graph

12. Import Policy

13. Provenance and Definition History

14. Typed Decoding

15. Semantic Validation

16. Schema Versioning

17. Secrets and Redaction

18. Diagnostics and Operability

19. Non-Goals for v1

20. Initial Internal Architecture

21. Possible Package Skeleton

22. Suggested Initial Implementation Sequence

Stage 1 — Core document model

Stage 2 — Strict composition

Stage 3 — Profile overlay

Stage 4 — Typed decoding

Stage 5 — Defaults and resolution hooks

Stage 6 — Semantic validation

Stage 7 — Introspection

23. Tests That Define Policy

24. Key Decisions Summary

20 KiB

Raw Blame History

8. `null`

11.1 `default` profile