diff --git a/README.md b/README.md index df6c8d9..47352d7 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,10 @@ imports, strictly composes their disjoint definitions, applies typed defaults, a can overlay one selected fixed profile and check an application schema version. It returns detached metadata and safe diagnostics. Applications may perform audited defaulting and normalization through a resolver, -then report domain errors and warnings through read-only semantic validation. -Overrides, secrets, and rendering are still under development. +then report domain errors and warnings through read-only semantic validation. Secret +tags and a path predicate drive deterministic redacted effective YAML. +Overrides, secret references/bootstrap, and value explanation are still under +development. ```go type Config struct { diff --git a/docs/config.md b/docs/config.md index 39ebd52..9557a77 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,9 +1,9 @@ # Configuration Input Contract This document owns ConfigKit's currently implemented source-input, YAML syntax, -schema, typed-decoding, loading, metadata, and ownership contracts. The parser, -owned document representation, and decoder remain private behind the public loading -lifecycle. +schema, typed-decoding, loading, metadata, redaction, rendering, and ownership +contracts. The parser, owned document representation, and decoder remain private +behind the public loading lifecycle. ## Sources @@ -42,7 +42,7 @@ The current lifecycle is: validate options → parse the selected source → traverse/compose enabled imports → build schema → check the optional application version → apply a selected profile → create and clone defaults → strictly decode → recorded application resolution - → semantic validation → snapshot configuration and metadata + → semantic validation → classify secrets → snapshot configuration and metadata ``` `File`, `Bytes`, and `Reader` each contribute the first base source with deterministic @@ -53,17 +53,18 @@ missing file remains an error; it is never interpreted as no input. The current `Options` boundary accepts a defaults factory, `VersionRule`, resolver, semantic validator, explicit path and glob imports, one selected profile, and -filesystem context. Overrides and dynamic secret classification return -`options.unavailable` because their lifecycle stages are not implemented. `NoSource` -rejects imports, active profiles, and filesystem context as invalid combinations. -Input containing a reserved `_configkit` mapping key is rejected while imports are -disabled. +filesystem context. It also accepts static secret tags and a dynamic path predicate. +Overrides return `options.unavailable` because their lifecycle stage is not +implemented. `NoSource` rejects imports, active profiles, and filesystem context as +invalid combinations. Input containing a reserved `_configkit` mapping key is +rejected while imports are disabled. Every source, options, schema, version, defaults, decode, resolution, or validation failure returns a nil result and a `*LoadError`. Successful results contain caller-owned `Config`, detached `Metadata`, a defensive source list, and defensive nonfatal diagnostics. Mutating the returned configuration or inspection records cannot alter ConfigKit's private -snapshot. ConfigKit performs no logging and exposes no effective writer yet. +snapshot. ConfigKit performs no logging. `Result.WriteEffective` exposes only the +frozen redacted projection. ### Application schema versions @@ -88,8 +89,7 @@ available source coordinates are retained. `History` records the Go-zero or appl default baseline followed by exact base or profile definitions, including equal-valued input. Profile replacement removes displaced descendants from effective metadata while retaining their base definition and the parent replacement operation -in history. -All metadata queries and returned nested records are detached. +in history. All metadata queries and returned nested records are detached. History is chronological: a path can have a synthetic default record, its exact base definition, a profile replacement, and a recorded resolution operation. A descendant @@ -108,6 +108,42 @@ operation. Reasons and dependency paths retain first occurrence order with dupli removed. Source records remain ordered as selected source, depth-first imports, then the selected profile. +### Secret classification and effective rendering + +Secret classification combines `configkit:"secret"` field tags with the optional +pure `Definition.Secret(Path) bool` predicate. A tagged or predicate-classified +container makes all descendants sensitive. ConfigKit evaluates final and historical +paths, so replacing or deleting a definition does not erase its classification. The +predicate can add sensitivity but cannot declassify a tagged path. + +Recorded resolution dependencies propagate sensitivity to their derived target. +Propagation reaches a fixed point, so record/map iteration order does not affect the +result and dependency cycles terminate. A dependency on a classified descendant or +on a container that contains one is treated conservatively as sensitive. Applications +must classify secret-producing transformations whose dependencies are not recorded. + +`Result.WriteEffective` writes the frozen resolved snapshot as deterministic redacted +YAML. Mappings sort by decoded key string, sequences retain order, indentation is two +spaces with noncompact sequence indentation, line endings are LF, the document has no +start or end marker, and output ends with a newline. Nil pointers/maps/slices remain +`null`; non-nil empty maps and sequences remain `{}` and `[]`. A secret root is +replaced by the quoted string `'[REDACTED]'`. Redacted output is intended for +inspection and is not promised to be reloadable. + +Rendering uses only a safe adapter tree built from the already completed canonical +typed projection. It does not invoke application YAML marshal hooks or text codecs; +all text-codec projection has succeeded before `Load` returns. Caller mutation of +`Result.Config` cannot change later output. Writer failures, including short writes, +are returned and may leave partial redacted output; internal dumper failures are +translated to a fixed safe error. There is no unredacted rendering option and +ConfigKit performs no implicit logging. + +`Result.Config` is the explicit trusted boundary and contains ordinary typed values, +including secrets. Metadata, histories, diagnostics, and load errors contain no raw +configuration values. Applications must still keep source labels, resolution +reasons, and diagnostic messages value-free because ConfigKit cannot infer and +sanitize secrets embedded in arbitrary application prose. + ### Recorded application resolution An optional `Definition.Resolve` callback runs after successful typed decoding. It @@ -356,7 +392,8 @@ or pointer-to-struct, or designates at most one string-keyed inline map for othe unmatched names. Ambiguous names, named inline fields, multiple inline maps, and unsupported or malformed tag options are schema errors. Embedded fields are ordinary named fields unless explicitly inlined. `configkit:"secret"` is the only current -ConfigKit field tag and records sensitivity for later lifecycle work. +ConfigKit field tag and marks its bound field or container for secret classification +and redacted inspection. Schema path traversal uses these same bindings. Explicit key segments address mappings, explicit index segments address sequences, and parsed numeric pointer diff --git a/docs/development.md b/docs/development.md index 75f8c44..1972038 100644 --- a/docs/development.md +++ b/docs/development.md @@ -4,8 +4,8 @@ This is the contributor entry point for ConfigKit. Read the task-specific owners below before making changes. The repository is in early implementation: its public loader currently covers strict selected sources, explicit path/glob import graphs, composition, selected profile overlays, typed defaults, schema-version checks, -recorded application resolution, -semantic validation, and metadata. The +recorded application resolution, semantic validation, layered metadata, secret +classification, and redacted effective rendering. The [component overview](internal/overview.md) owns the implementation inventory. ## Initial orientation diff --git a/docs/internal/extraction.md b/docs/internal/extraction.md index d3952dd..3eeb831 100644 --- a/docs/internal/extraction.md +++ b/docs/internal/extraction.md @@ -200,6 +200,18 @@ snapshot agreement, defensive nested copies, and concurrent load isolation. Valu rendering, redaction, explanation, commands, generated attribution, diffs, and digests remain assigned to their later owning work. +Effective-rendering implementation record (2026-09-06): ConfigKit adapts Narratio's +deterministic effective-output and side-effect-free inspection cases in +`classifySecretPaths`, `redactedDocument`, `Result.WriteEffective`, and the private +`internal/yaml.Write` adapter. Tests cover stable sorted mappings, preserved sequence +order, explicit indentation/line/document formatting, tagged containers, dynamic +current and historical paths, fixed-point dependency cycles, nil/empty distinctions, +resolved values, text-scalar snapshot reuse, application YAML-hook isolation, +non-disclosure through failures and record formatting, caller mutation, and writer +failures. Unlike Narratio's marshal/reparse path, ConfigKit renders only its owned +frozen tree and substitutes redaction before the upstream dumper boundary. Historical +value explanation and secret references remain assigned to later work. + ### Defaults, presence, and domain resolution Assigned across schema, decoding, resolution, validation, and consumer examples: diff --git a/docs/internal/overview.md b/docs/internal/overview.md index f7aa4c9..a25fb5d 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -30,11 +30,15 @@ module and package documentation boundary. Its implemented public primitives are explicitness queries, helper verification, structural mutation audit, coalesced intent, private before/after snapshots, and resolution metadata integration; and - post-resolution schema-aware report collection and validator mutation detection in - `validation.go`, completing the public callback lifecycle after composed input. + `validation.go`, completing the public callback lifecycle after composed input; +- fixed-point tag/predicate/dependency sensitivity classification in `secret.go`; and +- frozen-snapshot redaction and canonical effective output in `render.go`, with the + only upstream dumper integration isolated in `internal/yaml/write.go`. The [configuration contract](../config.md) owns implemented input, schema, decoding, loading lifecycle, path imports, composition, profile overlay, version checking, -recorded resolution, semantic validation, metadata, and ownership behavior. +recorded resolution, semantic validation, metadata, secret classification, redacted +effective rendering, and ownership behavior. The source-extraction inventory records the application foundations assigned to future implementation work: diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 9e6a1af..ff1970d 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -2,7 +2,7 @@ ## Status and ownership -Status: implementation in progress; Stages 1–14 are complete and later stages are +Status: implementation in progress; Stages 1–15 are complete and later stages are pending. Audience: a `gpt-5.6-terra` coding agent implementing one stage per prompt, in @@ -1092,7 +1092,36 @@ Acceptance: ## Stage 15 — Implement secret classification and effective rendering -Status: pending. +Status: complete. + +Completion evidence (2026-09-06): + +- Removed the dynamic-secret gate and combined descriptor `configkit:"secret"` roots + with a pure application path predicate over final and historical paths. Tagged + containers cover descendants, replaced/deleted definitions stay classified, and + resolution dependencies propagate sensitivity to a fixed point, including + container dependencies and cycles independent of record-map iteration order. +- Added `Result.WriteEffective` over the frozen resolved document. It clones and + redacts before crossing the adapter boundary, sorts mappings by decoded key, + preserves sequence order and nil/empty distinctions, and never invokes application + YAML hooks or text codecs during rendering. Caller mutation cannot change output; + there is no unredacted path or implicit logging. +- Added a private v4 dumper adapter with explicit two-space/noncompact-sequence + indentation, unlimited line width, Unicode, unique keys, block collections, + single-quote preference, LF endings, one document, and disabled start/end markers. + Secret roots render as the quoted string `'[REDACTED]'`; writer errors and short + writes propagate while internal dumper errors collapse to a fixed safe error. +- Tests cover tagged containers, dynamic current and overwritten paths, dependency + cycles, container dependencies, custom text scalars, application marshal-hook + isolation, resolved values, sorted/stable YAML, sequence order, nil/empty forms, + snapshot mutation isolation, writer failures, and canary non-disclosure through + parsing, decoding, returned hook errors, warnings, metadata, and ordinary/Go-syntax + record formatting. The + [current redaction contract](../config.md#secret-classification-and-effective-rendering) + owns the behavior. Historical-value explanation and secret references/bootstrap + remain later work. Verified `go test ./...`, `go test -race ./...`, `go vet ./...`, + `go build ./...`, `go mod tidy -diff`, and `git diff --check` with Go 1.26.7 on + linux/amd64. Read: feature [redaction](initial.md#provenance-diagnostics-and-redaction), [secret predicate](initial.md#schema-checking-and-secret-classification), result API. diff --git a/docs/roadmap/initial.md b/docs/roadmap/initial.md index 1f76a6b..ce8ba88 100644 --- a/docs/roadmap/initial.md +++ b/docs/roadmap/initial.md @@ -470,7 +470,8 @@ The [current application schema contract](../config.md#application-schema-bindin owns root/type support, field and tag binding, inlining, codec pairing, and schema-aware path rules. The [current defaults and typed-decoding contract](../config.md#defaults-and-typed-decoding) owns strict conversions, container application, presence, safe failures, and initial -origins. Secret handling and rendering remain target work below. +origins. Secret classification and effective rendering are owned by the +[current redaction contract](../config.md#secret-classification-and-effective-rendering). ## Recorded resolution and validation @@ -479,7 +480,7 @@ Application-owned recorded resolution is implemented. The explicitness queries, helpers and manual records, structural mutation auditing, coalescing, diagnostics, origins, history, and snapshot ownership. Layered input history is owned by the [current metadata contract](../config.md#layered-metadata); -redacted explanation and secret dependency propagation remain target work. +redacted historical-value explanation remains target work. Canonical typed projection, detached cloning, structural comparison, and publishing independently captured resolved results are implemented. The @@ -499,7 +500,9 @@ Safe diagnostic/provenance value records, canonical diagnostic rendering, and Default, base, profile, and resolution records are attached to the loading lifecycle; their source ordering, composite/synthetic origins, replacement history, and detached queries are owned by the [current metadata contract](../config.md#layered-metadata). -Safe historical values, redaction, and rendering remain target work below. +Secret classification and effective rendering are owned by the +[current redaction contract](../config.md#secret-classification-and-effective-rendering). +Safe historical values and explanation remain target work below. Diagnostics have stable codes, severity, semantic path when available, a safe message, primary location, and related locations. Sort deterministically by @@ -520,20 +523,6 @@ Applications likewise must supply value-free validation messages and resolution reasons. ConfigKit cannot sanitize secrets embedded in arbitrary application prose reliably. -Use `configkit:"secret"` on fields and an application path-classification hook -for dynamic entries. A secret container makes descendants secret. Secret -classification applies conservatively across current and historical definitions -of a path: replacing a secret does not declassify its history. It propagates from -recorded secret dependencies to derived outputs. Applications remain responsible -for marking secret-producing transformations whose dependencies are not recorded. - -Provide source reporting and redacted effective rendering in the initial target. -Render from a canonical projection of the final typed snapshot, including zero -values and null/empty distinctions supported by the schema. Redacted output is -for inspection and is not promised to be reloadable. ConfigKit renderers must -not expose raw secrets; returning typed configuration to the application is an -intentional trusted boundary. ConfigKit does not log configuration on its own. - Provide full safe historical-value explanation in the initial release, as defined [below](#historical-value-explanation). Defer config diffs and fingerprints. A future fingerprint design must specify canonicalization, secret handling, and disclosure risks before @@ -568,9 +557,9 @@ and verification against consumer models. `Load`, `Definition`, `Options`, `FilesystemContext`, `Override`, `VersionRule`, `Version`, `Validation`, and `Result` are declared. The current loader supports the -callback and path-import subset described by the [current contract](../config.md); -later override, secret, and rendering behavior below remains target work and -unavailable options fail explicitly rather than being ignored. +implemented boundary described by the [current contract](../config.md). Explicit +overrides, secret references/bootstrap, and historical-value explanation remain +target work; unavailable options fail explicitly rather than being ignored. ```go func Load[T any](source Source, options ...Options[T]) (*Result[T], error) @@ -863,8 +852,9 @@ potentially sensitive configured values. `VersionRule`, `Version`, and the version-checking use of `Validation` are implemented. The [current version contract](../config.md#application-schema-versions) owns absent/null/string handling, report locations, warnings, failure behavior, and -raw-error suppression. Secret classification and later composed-source placement -checks remain target work. +raw-error suppression. The +[current redaction contract](../config.md#secret-classification-and-effective-rendering) +owns secret classification. ```go type VersionRule struct { @@ -918,19 +908,15 @@ versions remain strings under ConfigKit's convention; adopting consumers with integer versions must migrate their syntax rather than enable a second scalar policy. -`Definition.Secret` is an optional pure path predicate for dynamic entries, -combined with secret field tags. It can add sensitivity but cannot override a -tag to declassify a value. It is applied to historical as well as final paths. -Recorded dependencies propagate sensitivity under the existing redaction rules; -applications own secret-producing transformations not described by those records. +`Definition.Secret` and `configkit:"secret"` behavior are implemented and owned by +the [current redaction contract](../config.md#secret-classification-and-effective-rendering). ### Results, errors, and inspection -`Result.Config`, `Metadata`, `Sources`, and `Diagnostics`, together with -`LoadError` and `WriteDiagnostics`, are implemented at the current loading boundary. +`Result.Config`, `Metadata`, `Sources`, `Diagnostics`, and `WriteEffective`, together +with `LoadError` and `WriteDiagnostics`, are implemented at the current loading boundary. The [current loading contract](../config.md#single-source-loading) and Go reference -comments own their behavior. Effective rendering and richer layered inspection -below remain target work. +comments own their behavior. Historical-value explanation below remains target work. ```go type Result[T any] struct { @@ -961,12 +947,9 @@ locations as described [above](#provenance-diagnostics-and-redaction). No raw parser error or unrestricted secret-bearing history is exposed. No automatic logging occurs. Inspection methods returning slices provide defensive copies. -`Config` is caller-owned typed data. Metadata and rendering use an independently -captured final snapshot, so later caller mutation cannot rewrite the recorded -result. `WriteEffective` emits deterministic redacted YAML for inspection, with -no unredacted switch. It returns output errors rather than a new configuration validation result. -Custom projection failures must already have been rejected before a load -succeeds. Output can be partial on writer failure. +Caller-owned configuration, the frozen result snapshot, secret classification, and +`WriteEffective` behavior are owned by the +[current redaction contract](../config.md#secret-classification-and-effective-rendering). An application can report errors and inspect a successful load without rebuilding diagnostic formatting. This example uses `errors` and `io` and the pipeline diff --git a/internal/yaml/write.go b/internal/yaml/write.go new file mode 100644 index 0000000..e6151d2 --- /dev/null +++ b/internal/yaml/write.go @@ -0,0 +1,103 @@ +package yaml + +import ( + "errors" + "io" + "strings" + + "go.yaml.in/yaml/v4" + + "gitea.maximumdirect.net/eric/ConfigKit/internal/document" +) + +var errWrite = errors.New("configkit: effective YAML rendering failed") + +// Write serializes one owned document tree with fully specified stable formatting. +func Write(output io.Writer, root *document.Node) error { + if output == nil || root == nil { + return errWrite + } + dumper, err := yaml.NewDumper(strictWriter{output}, + yaml.WithIndent(2), + yaml.WithCompactSeqIndent(false), + yaml.WithLineWidth(-1), + yaml.WithUnicode(true), + yaml.WithUniqueKeys(true), + yaml.WithCanonical(false), + yaml.WithLineBreak(yaml.LineBreakLN), + yaml.WithExplicitStart(false), + yaml.WithExplicitEnd(false), + yaml.WithFlowSimpleCollections(false), + yaml.WithQuotePreference(yaml.QuoteSingle), + yaml.WithAllDocuments(false), + ) + if err != nil { + return errWrite + } + if err := dumper.Dump(adapterNode(root)); err != nil { + return safeWriteError(err) + } + if err := dumper.Close(); err != nil { + return safeWriteError(err) + } + return nil +} + +func adapterNode(node *document.Node) *yaml.Node { + result := &yaml.Node{} + switch node.Kind { + case document.Mapping: + result.Kind, result.Tag = yaml.MappingNode, "!!map" + result.Content = make([]*yaml.Node, 0, len(node.Mapping)*2) + for _, entry := range node.Mapping { + result.Content = append(result.Content, + &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: entry.Key}, + adapterNode(entry.Value), + ) + } + case document.Sequence: + result.Kind, result.Tag = yaml.SequenceNode, "!!seq" + result.Content = make([]*yaml.Node, len(node.Sequence)) + for index, child := range node.Sequence { + result.Content[index] = adapterNode(child) + } + case document.String: + result.Kind, result.Tag, result.Value = yaml.ScalarNode, "!!str", node.Text + if node.Text == "[REDACTED]" { + result.Style = yaml.SingleQuotedStyle + } + case document.Boolean: + result.Kind, result.Tag, result.Value = yaml.ScalarNode, "!!bool", node.Text + case document.Number: + result.Kind, result.Value = yaml.ScalarNode, node.Text + result.Tag = "!!int" + if strings.ContainsAny(node.Text, ".eE") { + result.Tag = "!!float" + } + case document.Null: + result.Kind, result.Tag, result.Value = yaml.ScalarNode, "!!null", "null" + } + return result +} + +func safeWriteError(err error) error { + var dumpError *yaml.DumpError + if errors.As(err, &dumpError) && dumpError.Stage == yaml.WriterStage { + if cause := errors.Unwrap(dumpError); cause != nil { + return cause + } + } + return errWrite +} + +type strictWriter struct { + output io.Writer +} + +func (writer strictWriter) Write(data []byte) (int, error) { + written, err := writer.output.Write(data) + if err == nil && written != len(data) { + err = io.ErrShortWrite + } + return written, err +} diff --git a/internal/yaml/write_test.go b/internal/yaml/write_test.go new file mode 100644 index 0000000..6ac6cef --- /dev/null +++ b/internal/yaml/write_test.go @@ -0,0 +1,59 @@ +package yaml + +import ( + "bytes" + "errors" + "io" + "testing" + + "gitea.maximumdirect.net/eric/ConfigKit/internal/document" +) + +func TestWriteUsesStableExplicitFormatting(t *testing.T) { + t.Parallel() + root := &document.Node{Kind: document.Mapping, Mapping: []document.Entry{ + {Key: "empty_map", Value: &document.Node{Kind: document.Mapping, Mapping: []document.Entry{}}}, + {Key: "empty_sequence", Value: &document.Node{Kind: document.Sequence, Sequence: []*document.Node{}}}, + {Key: "number", Value: &document.Node{Kind: document.Number, Text: "1.25e+2"}}, + {Key: "redacted", Value: &document.Node{Kind: document.String, Text: "[REDACTED]"}}, + {Key: "sequence", Value: &document.Node{Kind: document.Sequence, Sequence: []*document.Node{ + {Kind: document.String, Text: "first"}, + {Kind: document.String, Text: "true"}, + }}}, + {Key: "nil_value", Value: &document.Node{Kind: document.Null}}, + }} + var output bytes.Buffer + if err := Write(&output, root); err != nil { + t.Fatal(err) + } + want := "empty_map: {}\nempty_sequence: []\nnumber: 1.25e+2\nredacted: '[REDACTED]'\nsequence:\n - first\n - 'true'\nnil_value: null\n" + if output.String() != want { + t.Fatalf("output:\n%s\nwant:\n%s", output.String(), want) + } +} + +func TestWritePropagatesWriterErrorsAndShortWrites(t *testing.T) { + t.Parallel() + root := &document.Node{Kind: document.Mapping, Mapping: []document.Entry{{Key: "value", Value: &document.Node{Kind: document.String, Text: "safe"}}}} + want := errors.New("write failed") + if err := Write(failingWriter{err: want}, root); !errors.Is(err, want) { + t.Fatalf("writer error = %v", err) + } + if err := Write(shortOutput{}, root); !errors.Is(err, io.ErrShortWrite) { + t.Fatalf("short writer error = %v", err) + } +} + +type failingWriter struct { + err error +} + +func (writer failingWriter) Write(data []byte) (int, error) { + return min(2, len(data)), writer.err +} + +type shortOutput struct{} + +func (shortOutput) Write(data []byte) (int, error) { + return max(0, len(data)-1), nil +} diff --git a/load.go b/load.go index 0b394ac..9e97567 100644 --- a/load.go +++ b/load.go @@ -7,6 +7,7 @@ import ( ) // Definition contains reusable application rules for a configuration type. +// Secret is a pure predicate that can add sensitivity to semantic paths. // Definitions should be treated as read-only after being shared. type Definition[T any] struct { Defaults func() T @@ -68,6 +69,7 @@ type Result[T any] struct { diagnostics []Diagnostic final *document.Node resolution []resolutionSnapshot + secrets map[string]bool } // Metadata returns detached effective presence, origin, and history records. @@ -213,6 +215,7 @@ func Load[T any](source Source, supplied ...Options[T]) (*Result[T], error) { return nil, accumulator.loadError() } } + secrets := classifySecretPaths(schema, options.Definition.Secret, metadata, final) return &Result[T]{ Config: configuration, metadata: metadata, @@ -220,6 +223,7 @@ func Load[T any](source Source, supplied ...Options[T]) (*Result[T], error) { diagnostics: canonicalDiagnostics(accumulator.diagnostics), final: document.Clone(final), resolution: cloneResolutionSnapshots(resolutionSnapshots), + secrets: secrets, }, nil } @@ -233,9 +237,6 @@ func selectOptions[T any](source Source, supplied []Options[T]) (Options[T], []D } var diagnostics []Diagnostic definition := options.Definition - if definition.Secret != nil { - diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsUnavailable, "dynamic secret classification is not available in this release boundary")) - } if !validProfileName(options.Profile) { diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsInvalid, "profile name must contain only ASCII letters, digits, underscore, or hyphen")) } diff --git a/load_test.go b/load_test.go index 0ed9529..b7169a2 100644 --- a/load_test.go +++ b/load_test.go @@ -222,7 +222,6 @@ func TestLoadRejectsUnavailableOptionsAndDisabledDirectives(t *testing.T) { code Code }{ {name: "multiple options", source: validSource(), options: []Options[config]{{}, {}}, code: CodeOptionsInvalid}, - {name: "secret classifier", source: validSource(), options: []Options[config]{{Definition: Definition[config]{Secret: func(Path) bool { return false }}}}, code: CodeOptionsUnavailable}, {name: "imports without filesystem context", source: validSource(), options: []Options[config]{{Imports: true}}, code: CodeOptionsInvalid}, {name: "incomplete filesystem", source: validSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsInvalid}, {name: "overrides", source: validSource(), options: []Options[config]{{Overrides: []Override{{Path: Keys("name"), Value: "other", Flag: "--name"}}}}, code: CodeOptionsUnavailable}, diff --git a/render.go b/render.go new file mode 100644 index 0000000..d1449dc --- /dev/null +++ b/render.go @@ -0,0 +1,70 @@ +package configkit + +import ( + "errors" + "io" + "slices" + "strings" + + "gitea.maximumdirect.net/eric/ConfigKit/internal/document" + yamladapter "gitea.maximumdirect.net/eric/ConfigKit/internal/yaml" +) + +var errEffectiveUnavailable = errors.New("configkit: effective configuration is unavailable") + +// WriteEffective writes deterministic redacted YAML from the frozen resolved +// snapshot. The output is intended for inspection and is not promised reloadable. +// Writer errors, including short writes, are returned and may leave partial output. +func (r *Result[T]) WriteEffective(output io.Writer) error { + if r == nil || r.final == nil || output == nil { + return errEffectiveUnavailable + } + safe := redactedDocument(r.final, r.secrets, Path{}) + return yamladapter.Write(output, safe) +} + +func redactedDocument(node *document.Node, secret map[string]bool, path Path) *document.Node { + if node == nil { + return nil + } + if pathIsSecret(secret, path) { + return &document.Node{Kind: document.String, Text: "[REDACTED]"} + } + result := *node + switch node.Kind { + case document.Mapping: + result.Mapping = make([]document.Entry, len(node.Mapping)) + copy(result.Mapping, node.Mapping) + slices.SortFunc(result.Mapping, func(left, right document.Entry) int { + return strings.Compare(left.Key, right.Key) + }) + for index := range result.Mapping { + entry := &result.Mapping[index] + entry.Value = redactedDocument(entry.Value, secret, path.Key(entry.Key)) + } + case document.Sequence: + result.Sequence = make([]*document.Node, len(node.Sequence)) + for index, child := range node.Sequence { + result.Sequence[index] = redactedDocument(child, secret, path.Index(index)) + } + } + return &result +} + +func pathIsSecret(secret map[string]bool, path Path) bool { + if secret[""] { + return true + } + pointer := path.String() + for pointer != "" { + if secret[pointer] { + return true + } + position := strings.LastIndexByte(pointer, '/') + if position <= 0 { + break + } + pointer = pointer[:position] + } + return false +} diff --git a/render_test.go b/render_test.go new file mode 100644 index 0000000..ea5138c --- /dev/null +++ b/render_test.go @@ -0,0 +1,177 @@ +package configkit + +import ( + "bytes" + "errors" + "fmt" + "io" + "strings" + "sync/atomic" + "testing" +) + +const ( + taggedCanary = "TAGGED_CANARY_7c65" + dynamicCanary = "DYNAMIC_CANARY_6a12" + historicalCanary = "HISTORICAL_CANARY_91ef" + derivedACanary = "DERIVED_A_CANARY_4c20" + derivedBCanary = "DERIVED_B_CANARY_831d" +) + +var renderTextMarshalCalls atomic.Int64 +var renderYAMLMarshalCalls atomic.Int64 + +type renderText string + +func (value renderText) MarshalText() ([]byte, error) { + renderTextMarshalCalls.Add(1) + return []byte(value), nil +} + +func (value *renderText) UnmarshalText(text []byte) error { + *value = renderText(text) + return nil +} + +type renderCredentials struct { + Token string `yaml:"token"` + Nested []string `yaml:"nested"` +} + +type renderConfig struct { + Credentials renderCredentials `yaml:"credentials" configkit:"secret"` + Custom renderText `yaml:"custom"` + DerivedA string `yaml:"derived_a"` + DerivedB string `yaml:"derived_b"` + Dynamic any `yaml:"dynamic"` + EmptyMap map[string]string `yaml:"empty_map"` + EmptySequence []string `yaml:"empty_sequence"` + Labels map[string]string `yaml:"labels"` + NilMap map[string]string `yaml:"nil_map"` + NilSequence []string `yaml:"nil_sequence"` + Sequence []string `yaml:"sequence"` +} + +func (renderConfig) MarshalYAML() (any, error) { + renderYAMLMarshalCalls.Add(1) + return map[string]string{"forbidden": "application YAML hook ran"}, nil +} + +func TestWriteEffectiveRedactsTagsDynamicHistoryAndDependencyCycles(t *testing.T) { + renderTextMarshalCalls.Store(0) + renderYAMLMarshalCalls.Store(0) + directory := t.TempDir() + mainPath := writeImportFile(t, directory, "main.yml", `credentials: + token: TAGGED_CANARY_7c65 + nested: [nested-secret] +custom: custom value +dynamic: {retired: HISTORICAL_CANARY_91ef} +empty_map: {} +empty_sequence: [] +labels: {hidden: DYNAMIC_CANARY_6a12, public: visible} +sequence: [second, first] +`) + writeImportFile(t, directory, "profiles/testing.yml", "dynamic: selected\n") + definition := Definition[renderConfig]{ + Secret: func(path Path) bool { + return path.String() == "/labels/hidden" || path.String() == "/dynamic/retired" + }, + Resolve: func(configuration *renderConfig, resolution *Resolution) error { + Normalize(resolution, Keys("derived_a"), &configuration.DerivedA, derivedACanary, + "derived through cycle", Keys("derived_b")) + Normalize(resolution, Keys("derived_b"), &configuration.DerivedB, derivedBCanary, + "derived from retired secret", Keys("derived_a"), Keys("dynamic")) + resolution.Warning(Keys("derived_a"), "derived output is configured") + return nil + }, + } + result, err := Load[renderConfig](File(mainPath), Options[renderConfig]{Definition: definition, Profile: "testing"}) + if err != nil { + t.Fatal(err) + } + if result.Config.Credentials.Token != taggedCanary || result.Config.Labels["hidden"] != dynamicCanary || + result.Config.DerivedA != derivedACanary || result.Config.DerivedB != derivedBCanary { + t.Fatal("typed configuration did not retain trusted values") + } + if !result.secrets["/dynamic/retired"] || + !result.secrets["/derived_a"] || !result.secrets["/derived_b"] { + t.Fatalf("secret closure = %#v", result.secrets) + } + if renderYAMLMarshalCalls.Load() != 0 { + t.Fatal("loading called an application YAML marshal hook") + } + textCalls := renderTextMarshalCalls.Load() + + var output bytes.Buffer + if err := result.WriteEffective(&output); err != nil { + t.Fatal(err) + } + want := "credentials: '[REDACTED]'\ncustom: custom value\nderived_a: '[REDACTED]'\nderived_b: '[REDACTED]'\ndynamic: selected\nempty_map: {}\nempty_sequence: []\nlabels:\n hidden: '[REDACTED]'\n public: visible\nnil_map: null\nnil_sequence: null\nsequence:\n - second\n - first\n" + if output.String() != want { + t.Fatalf("effective output:\n%s\nwant:\n%s", output.String(), want) + } + if renderTextMarshalCalls.Load() != textCalls || renderYAMLMarshalCalls.Load() != 0 { + t.Fatal("rendering called an application marshal hook") + } + for _, canary := range []string{taggedCanary, dynamicCanary, historicalCanary, derivedACanary, derivedBCanary, "nested-secret"} { + if strings.Contains(output.String(), canary) { + t.Fatalf("effective output contains %q", canary) + } + formatted := fmt.Sprintf("%v %#v %v %#v", result.Metadata(), result.Metadata(), result.Diagnostics(), result.Diagnostics()) + if strings.Contains(formatted, canary) { + t.Fatalf("metadata formatting contains %q", canary) + } + } + + beforeMutation := output.String() + result.Config.Credentials.Token = "MUTATED_SECRET" + result.Config.Labels["hidden"] = "MUTATED_DYNAMIC_SECRET" + result.Config.Sequence[0] = "mutated" + output.Reset() + if err := result.WriteEffective(&output); err != nil || output.String() != beforeMutation { + t.Fatalf("frozen output after caller mutation = %q, %v", output.String(), err) + } + + wantWriteError := errors.New("write failed") + if err := result.WriteEffective(errorWriter{err: wantWriteError}); !errors.Is(err, wantWriteError) { + t.Fatalf("writer error = %v", err) + } + if err := result.WriteEffective(shortWriter{}); !errors.Is(err, io.ErrShortWrite) { + t.Fatalf("short writer error = %v", err) + } + var nilResult *Result[renderConfig] + if err := nilResult.WriteEffective(io.Discard); !errors.Is(err, errEffectiveUnavailable) { + t.Fatalf("nil result error = %v", err) + } +} + +func TestSecretValuesStayOutOfLoadFailuresAndWarnings(t *testing.T) { + tests := []struct { + name string + input string + definition Definition[renderConfig] + }{ + {name: "parse", input: "credentials: [" + taggedCanary + "\n"}, + {name: "decode", input: "credentials: " + taggedCanary + "\n"}, + {name: "returned hook error", input: "credentials: {token: " + taggedCanary + "}\n", definition: Definition[renderConfig]{ + Resolve: func(*renderConfig, *Resolution) error { return errors.New(taggedCanary) }, + }}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + test.definition.Secret = func(path Path) bool { return path.String() == "/credentials" } + result, err := Load[renderConfig](Bytes("secret-failure.yml", []byte(test.input)), Options[renderConfig]{Definition: test.definition}) + if result != nil || err == nil { + t.Fatalf("result, error = %#v, %v", result, err) + } + var loadError *LoadError + if !errors.As(err, &loadError) { + t.Fatalf("error type = %T", err) + } + formatted := fmt.Sprintf("%v %#v %v %#v %v %#v", err, err, loadError.Diagnostics(), loadError.Diagnostics(), loadError.Sources(), loadError.Sources()) + if strings.Contains(formatted, taggedCanary) { + t.Fatalf("failure formatting disclosed a secret: %s", formatted) + } + }) + } +} diff --git a/secret.go b/secret.go new file mode 100644 index 0000000..8b2d356 --- /dev/null +++ b/secret.go @@ -0,0 +1,124 @@ +package configkit + +import "gitea.maximumdirect.net/eric/ConfigKit/internal/document" + +type secretDependency struct { + target Path + dependency Path +} + +func classifySecretPaths(schema *schemaDescriptor, predicate func(Path) bool, metadata Metadata, final *document.Node) map[string]bool { + paths := make(map[string]Path) + finalPaths := make(map[string]*document.Node) + indexDocument(final, Path{}, finalPaths) + for pointer := range finalPaths { + addSecretPath(paths, pointer) + } + var dependencies []secretDependency + for pointer, history := range metadata.history { + addSecretPath(paths, pointer) + for _, change := range history { + paths[change.Path.String()] = change.Path.clone() + for _, dependency := range change.Dependencies { + paths[dependency.String()] = dependency.clone() + dependencies = append(dependencies, secretDependency{ + target: change.Path.clone(), dependency: dependency.clone(), + }) + } + } + } + + secret := make(map[string]bool) + for pointer, path := range paths { + if root, ok := schemaSecretRoot(schema, path); ok { + secret[root.String()] = true + } + if predicate != nil && predicate(path.clone()) { + secret[pointer] = true + } + } + for { + changed := false + for _, dependency := range dependencies { + if secretIntersectsPath(paths, secret, dependency.dependency) && !secret[dependency.target.String()] { + secret[dependency.target.String()] = true + changed = true + } + } + if !changed { + return secret + } + } +} + +func addSecretPath(paths map[string]Path, pointer string) { + if path, err := ParsePath(pointer); err == nil { + paths[pointer] = path + } +} + +func secretIntersectsPath(paths map[string]Path, secret map[string]bool, candidate Path) bool { + for pointer := range secret { + classified, ok := paths[pointer] + if !ok { + continue + } + if pathContains(classified, candidate) || pathContains(candidate, classified) { + return true + } + } + return false +} + +func schemaSecretRoot(schema *schemaDescriptor, path Path) (Path, bool) { + if schema == nil || schema.root == nil || !path.valid() { + return Path{}, false + } + current := schema.root + walked := Path{} + for _, segment := range path.segments { + walked.segments = append(walked.segments, segment) + for current.kind == descriptorPointer { + current = current.element + } + switch current.kind { + case descriptorStruct: + if segment.kind == pathSegmentIndex { + return Path{}, false + } + position, ok := current.fieldByName[segment.key] + var field *fieldDescriptor + if ok { + field = ¤t.fields[position] + } else { + field = current.inlineMap + } + if field == nil { + return Path{}, false + } + if field.secret { + return walked.clone(), true + } + if field.inlineMap { + current = field.descriptor.element + } else { + current = field.descriptor + } + case descriptorMap: + if segment.kind == pathSegmentIndex { + return Path{}, false + } + current = current.element + case descriptorSlice, descriptorArray: + if _, ok := schemaIndex(segment); !ok { + return Path{}, false + } + current = current.element + case descriptorDynamic: + return Path{}, false + default: + return Path{}, false + } + } + return Path{}, false +}