ConfigKit/docs/internal/extraction.md

# Source Extraction Inventory

This inventory records reusable implementation evidence from sibling applications.
Those checkouts are review inputs only: ConfigKit does not import them, and its
build and tests must not require them. ConfigKit's current contracts and roadmap
govern every adaptation.

## Inspected revisions

| Application | Revision | Worktree | Role |
| --- | --- | --- | --- |
| Narratio | `b68435128be1e24b94fc7ec259d6e9093669fd99` | Clean `main` tracking `origin/main` | Primary parsing, composition, import, profile, presence, defaults, resolution, and inspection source |
| WeatherReporter | `5d306a202471e10fe98923a1df27d0b0fce5c666` | Clean `main` tracking `origin/main` | Secrets bootstrap and simple-consumer/defaults/options evidence |
| Notarius | `77ae132e5609916da40dcd4fcc749c000a63cbac` | Clean `main` tracking `origin/main` | Canonical binding, string-version, and dynamic-option migration evidence |
| Seriatim | `0fc92f364316c874d366b00cdc884a33e1900b25` | Clean `main` tracking `origin/main` | Independent ordered speaker/autocorrect rule evidence |

These revisions were inspected on 2026-09-06. Paths below are relative to each
application repository. Record later source revisions and adaptations here when
the owning implementation work extracts a symbol or case.

## Adaptation rules

- Copy and generalize only the smallest applicable mechanism or behavioral case.
- Replace upstream YAML nodes, application types, commands, logging, digests, and
  provider registries with ConfigKit-owned boundaries.
- Preserve strictness, deterministic behavior, and useful failure coverage while
  changing expectations that conflict with ConfigKit's accepted contracts.
- Use sanitized local fixtures only. Audio, transcripts, credentials, private
  configuration, and whole application package trees are application-only.
- Record the exact copied/generalized symbols and tests in the owning section when
  extraction occurs. Assignment prose records the original work boundary; dated
  implementation records state what was actually generalized.

## Narratio evidence

### Source parsing and strict input

Assigned to the YAML/source implementation:

- Production: `internal/config/load.go` functions `decodeStrictYAML`,
  `decodeStrictYAMLFromReader`, and `LoadSessionBytesWithOptions`; and
  `internal/config/composition.go` functions `parseCompositionDocument`,
  `parseCompositionBytes`, and `buildCompositionNode`.
- Cases: `TestLoadersRejectTrailingYAMLDocuments`,
  `TestStrictYAMLRejectsParserExposedEmptyTrailingDocument`, and
  `TestStrictYAMLAcceptsSingleDocumentAndRejectsUnknownFields` in
  `internal/config/loader_strict_test.go`; plus duplicate-key, alias, non-mapping,
  malformed-input, scalar spelling, presence, declaration-order, and source-label
  coverage in `TestParseCompositionDocumentRetainsPresenceOwnershipAndDeclarationOrder`
  and `TestParseCompositionDocumentRejectsAmbiguousOrMalformedYAML`.
- Preserve: labeled reader/byte input, strict unknown-field intent, rejection of
  trailing documents (including an empty second document), duplicate detection,
  and source-aware failures.
- Adapt: translate v4 parser nodes into ConfigKit's private document tree, apply
  ConfigKit's scalar grammar and limits, and replace raw parser errors with safe
  diagnostics. Narratio's application decoders and YAML node types are not copied.

Implementation record (2026-09-06): the labeled byte/reader ownership and strict
single-document cases above were generalized into `source.go`, `parse.go`, and
`internal/yaml`. No production line was copied verbatim. ConfigKit adds byte and
tree ceilings, shared atomic reader claims, regular-file checks, exact numeric
lexemes, parser-depth calibration, ConfigKit-owned locations, safe diagnostic
codes, and an AST import-boundary regression. Application typed decoding, parser
errors, YAML node types, and session-specific loaders were not retained.

### Strict composition

Assigned to document composition:

- Production: `mergeAdditiveComposition`, `mergeAdditiveCompositions`,
  `validateAdditiveClaims`, `appendCompositionClaims`, `mergeAdditiveNodes`,
  `newCompositionConflict`, and cloning/path/source helpers in
  `internal/config/composition.go`.
- Cases: `TestMergeAdditiveCompositionJoinsOnlyDisjointMappings`,
  `TestMergeAdditiveCompositionRejectsEveryDuplicateClass`, and
  `TestMergeAdditiveCompositionReportsAllClaimingSources`.
- Preserve: recursive joining of disjoint mappings, collisions for every overlapping
  non-mapping value (including equal values and sequences), all claimant sources,
  deterministic traversal, and nonmutation of inputs.
- Adapt: empty mappings follow ConfigKit's no-child operation rule; paths use the
  ConfigKit semantic path model; both precise locations and composite origins are
  retained. Default values never participate in strict composition.

Implementation record (2026-09-06): `composeDocuments`, its pre-order claim
collection, conflict construction, and recursive mapping join in `compose.go` adapt
the assigned production symbols. `TestComposeDocumentsJoinsDisjointMappingsAndTracksClaims`,
`TestComposeDocumentsRejectsEveryPresentNonMappingOverlap`, and
`TestComposeDocumentsReportsAllClaimantsAtExactLocations` adapt the assigned cases.
The intentional empty-mapping difference is covered separately: empty mappings retain
container provenance but contribute no child claims. No other semantic deviation
from the extracted additive composer was required.

### Imports

Assigned to explicit import traversal and glob handling:

- Production: `loadPipelineCompositionSources`,
  `splitPipelineCompositionEnvelope`, `loadPipelineImports`, and related ownership
  structures in `internal/config/pipeline_composition.go`.
- Cases: `TestSplitPipelineCompositionEnvelopePreservesInputAndDeclarationOrder`,
  `TestSplitPipelineCompositionEnvelopeRejectsMalformedShapesWithoutMutation`,
  `TestLoadPipelineCompositionImportsDisjointFieldsAndTracksOwnership`,
  `TestLoadPipelineCompositionMergesDisjointKeyedEntries`,
  `TestLoadPipelineCompositionRejectsBaseConflictsWithAllSources`,
  `TestLoadPipelineCompositionRejectsUnsafeOrInvalidImports`,
  `TestLoadPipelineCompositionRejectsSameFileAliasesAndImportedComposition`,
  `TestLoadPipelineCompositionReportsImportedParseAndSchemaSources`, and
  `TestLoadPipelineCompositionKeepsRelativePathsRootBased`.
- Preserve: explicit declarations, declaration-order traversal, disjoint
  composition, conflict attribution, malformed/unreadable target failures, directive
  removal, and imported parse/schema source labels.
- Adapt: replace the root-only `composition` envelope with recursive
  `_configkit.imports`; add path and glob declarations, bounded depth-first graph
  traversal, canonical containment, cycle/diamond routes, and fixed source IDs.
  Canonical paths—not `os.SameFile`—define identity, so hard links remain distinct.

Path-import implementation record (2026-09-06): `splitImportDirectives`,
`parseImportDeclaration` (now in `directives.go`), `importFilesystem` (now in
`filesystem.go`), and `importTraversal.visitPath`/`loadImportedDocuments` in
`imports.go` adapt the assigned envelope, loading, source,
and failure mechanics. The corresponding ConfigKit tests adapt the assigned
declaration-order, disjoint composition, keyed-entry, all-source conflict, unsafe
target, alias, imported parse/schema, and relative-resolution cases. ConfigKit adds
recursive bounded DFS, `os.OpenRoot` confinement, canonical symlink identity,
cycle/diamond route locations, optional-path semantics, aggregate budgets, and
fragment version-placement enforcement. That checkpoint parsed glob syntax behind an
explicit unsupported gate that the subsequent glob implementation removed; hard
links intentionally replace Narratio's same-file rejection with path identity.

Glob-import implementation record (2026-09-06): ConfigKit retains the extracted
declaration order, import failures, composition, and identity cases while adding the
new `expandImportGlob`, `readGlobMatches`, and pattern-validation mechanisms in
`glob.go`. Focused cases cover lexical ordering, multiple declarations, overlap,
directories, hard links, contained and escaping symlinks, unmatched declarations,
malformed and recursive patterns, escaped metacharacters, character classes,
permission failures, explicit `conf.d` use, and entry/candidate budgets. This work
has no Narratio production equivalent; it uses the selected standard-library
`path.Match` and bounded `File.ReadDir` adaptation rather than its flat loader.

### Profiles and overlay

Assigned to selected-profile overlay:

- Production: `mergeOverlayComposition`, `mergeOverlayNodes`,
  `selectPipelineProfile`, `loadPipelineProfileOverlays`, and
  `loadedProfileOverlay`.
- Cases: `TestLoadPipelineProfilesSelectDefaultAndExplicitOverlay`,
  `TestLoadPipelineProfileSelectionErrors`,
  `TestPipelineProfileNamesRejectControlCharacters`,
  `TestLoadPipelineProfilesValidateEveryDeclaredOverlay`,
  `TestLoadPipelineProfileOverlaySemantics`,
  `TestLoadPipelineProfileRejectsNullAndKindChanges`, and
  `TestLoadPipelineProfilePairResolvesIndependentEffectivePipelines`.
- Preserve: recursive mapping overlay, atomic scalar/sequence replacement,
  deterministic selection failures, source ownership, and independent results.
- Adapt: load only the selected fixed profile path; lowercase `default` means no
  overlay. ConfigKit permits null and cross-kind replacement and does not support
  declared defaults, inheritance, arbitrary profile paths, or validating every
  unselected overlay.

Implementation record (2026-09-06): `overlayDocuments`, `mergeOverlayMappings`,
`loadProfileDocument`, and `profileDirectiveDiagnostics` adapt the assigned merge,
selection, loading, and ownership mechanics. `TestOverlayDocumentsRecursesMappingsAndReplacesAtomicSubtrees`,
`TestLoadAppliesSelectedProfileAfterBaseAndBeforeDefaults`, profile selection/error,
directive/version, containment, and budget cases adapt the assigned tests. ConfigKit
uses exactly one fixed selected `.yml` source, permits null and cross-kind replacement,
retains displaced descendant history, rejects any profile directive or version-path
contribution, and counts the profile in aggregate budgets. Narratio's declared
default, multi-overlay validation, inheritance-like application conventions, and
null/kind restrictions were not retained.

### Effective inspection and provenance

Assigned to layered metadata, redaction, rendering, and explanation:

- Production: `EffectivePipelineValues`, `EffectivePipelineSources`,
  `MarshalEffectivePipeline`, `effectivePipelineFieldPaths`,
  `effectivePipelineOwners`, and composition `semanticRecords`/
  `compactSemanticRecords`.
- Cases: `TestCompositionCanonicalOutputsAreDeterministic`,
  `TestEffectivePipelineValuesIgnoreEquivalentSourceLayout`,
  `TestConfigValidateAndShowPipelineOnlyAreSideEffectFree`,
  `TestConfigSourcesReportsStableOwnership`,
  `TestConfigSourcesReportsCanonicalFamilyAndPublishOrigins`,
  `TestConfigSourcesPreservesProfileOwnershipThroughFamilyExpansion`, and
  `TestConfigSourcesReportsRootImportProfileAndDefaultOwnership`.
- Preserve: deterministic effective projection, stable logical paths, source
  ownership, side-effect-free inspection, and domain-attribution evidence.
- Adapt: project directly from ConfigKit-owned typed values without YAML round trips;
  add line/column locations, composite origins, complete operation history, detached
  records, and redaction. Narratio commands, diff/digest APIs, generated artifact
  attribution, campaign logic, and party expansion remain application-owned.

Layered-provenance implementation record (2026-09-06): ConfigKit integrated its
previously extracted source claims, overlay displacement records, typed baseline,
and audited resolution snapshots in `load_metadata.go`. The representative
`TestLayeredProvenanceAgreesWithResolvedSnapshot` adapts Narratio's stable ownership,
profile ownership, deterministic effective projection, and side-effect-free
inspection cases across one default/import/profile/resolution workflow. ConfigKit
adds chronological descendant history for parent and whole-sequence operations,
composite empty-mapping origins, canonical resolution reasons/dependencies, private
snapshot agreement, defensive nested copies, and concurrent load isolation. Value
That checkpoint deliberately left rendering, redaction, explanation, commands,
generated attribution, diffs, and digests outside its scope.

Effective-rendering implementation record (2026-09-06): ConfigKit adapts Narratio's
deterministic effective-output and side-effect-free inspection cases in
`classifySecretPaths`, `redactedDocument`, `Result.WriteEffective`, and the private
`internal/yaml.Write` adapter. Tests cover stable sorted mappings, preserved sequence
order, explicit indentation/line/document formatting, tagged containers, dynamic
current and historical paths, fixed-point dependency cycles, nil/empty distinctions,
resolved values, text-scalar snapshot reuse, application YAML-hook isolation,
non-disclosure through failures and record formatting, caller mutation, and writer
failures. Unlike Narratio's marshal/reparse path, ConfigKit renders only its owned
frozen tree and substitutes redaction before the upstream dumper boundary. Historical
value explanation and secret references were implemented at their later owning
boundaries and are exercised by the consumer startup example below.

### Defaults, presence, and domain resolution

Assigned across schema, decoding, resolution, validation, and consumer examples:

- Production: `NormalizeConfig.UnmarshalYAML`, `outputPathWasSet`,
  `applyPipelineDefaults` and its component default helpers, `Resolve`,
  `ResolveLoadedPipelineCampaign`, `mergeCampaignSession`, `selectStableInput`, and
  application-supplied byte loading in `internal/app/config_loader.go`.
- Cases: `TestNormalizeLoadAndValidate`,
  `TestRepresentativeLoadDefaultsAndValidation`,
  `TestMarshalEffectivePipelineRendersStablePublicConfiguration`,
  `TestPipelineCampaignRegistryStrictDecode`,
  `TestCampaignSessionMergeFillsStableInputs`,
  `TestCampaignSessionMergeSessionOverridesStableInputs`,
  `TestLoadWithSessionOptionsCarriesExplicitProfilePresence`, and the maintained
  extraction-example coverage in `TestMaintainedExtractionExamplesPreservePublishedContracts`.
- Preserve: fresh defaults, explicit-presence-sensitive domain decisions, strict
  nested types, stable input selection, application-supplied labeled bytes, and
  validation after resolution.
- Adapt: presence comes from metadata rather than custom YAML hooks or pointer/zero
  guesses; explicit empty and null values are retained; defaults are cloned and
  application-owned; every domain mutation is recorded. Campaign/session fallback,
  search paths, fetching, storage, and concrete default values remain in Narratio.

Implementation record (2026-09-06): ConfigKit generalized Narratio's fresh-default,
strict nested-type, and explicit-presence cases into `decode.go` and
`decode_test.go`. The decoder applies the owned document tree over a detached typed
baseline, treats mappings recursively and sequences atomically, preserves explicit
false/zero/empty/null, rejects unknown nested fields, and returns private presence
and default/input origin artifacts. Unlike Narratio's custom YAML hooks and
pointer/zero inference, presence comes directly from the parsed tree and equal-valued
input remains explicit. ConfigKit also adds destination-width numeric checks,
dynamic `int64`/`float64`, exact arrays, duration strings, paired text codecs, safe
multi-error collection, and repeated-result isolation. Resolution, validation,
consumer defaults, and all campaign/session algorithms remain assigned to later
work.

Lifecycle integration record (2026-09-06): the public single-source `Load` boundary
combines the already extracted labeled-source and strict-decoding mechanisms with
Narratio's fresh-default and presence expectations. ConfigKit adds an application-
defined pre-decode string-version check, defaults-only `NoSource` operation,
detached base/default metadata and source reports, and explicit gates for every
later lifecycle option. No application loader or production line was copied.
Narratio's resolution, validation, import/profile, campaign/session, and command
behavior remains assigned to its later owning work.

Resolution implementation record (2026-09-06): ConfigKit adapted the intent of
Narratio's `Resolve`, `applyPipelineDefaults`, component default helpers,
`mergeCampaignSession`, and `selectStableInput` into a generic recorded resolver in
`resolution.go`. The synthetic timeout/artifact cases preserve explicit-input
decisions, derived dependencies, normalization, and map copy/writeback without
copying campaign/session algorithms or concrete defaults. ConfigKit adds structural
mutation auditing, container coverage, helper/path consistency checks, equal-value
intent, coalesced repeated writes, private before/after snapshots, deterministic
record ordering, and safe hook diagnostics. Semantic validation remains assigned to
the next lifecycle work.

Validation implementation record (2026-09-06): ConfigKit generalized Narratio's
`TestNormalizeLoadAndValidate`, `TestRepresentativeLoadDefaultsAndValidation`,
strict registry decoding, and maintained extraction-example expectations into the
post-resolution `Validation` lifecycle and synthetic consumer examples. ConfigKit
keeps domain rules application-owned, resolves schema-valid missing/dynamic report
paths to nearest input locations, collects independent warnings/errors, and audits
the full typed projection to reject validator mutation. Live resources, registries,
campaign/session logic, commands, and application runtime initialization were not
extracted.

### Secrets bootstrap

Assigned to the explicit bootstrap operation:

- Production: `loadSecretsFromConfigWithDependencies`, `validateSecretDirectory`,
  `validateSecretFile`, `readAPIKeyFile`, and
  `internal/fileops/read_bounded.go` function `ReadFileBounded`.
- Cases: `TestLoadSecretsFromConfigLoadsValidFiles`,
  `TestLoadSecretsFromConfigPreservesExistingEnv`,
  `TestLoadSecretsFromConfigRelativeDirUsesCWD`,
  `TestLoadSecretsFromConfigMissingDirFails`,
  `TestLoadSecretsFromConfigUnreadableValidEntryFails`,
  `TestLoadSecretsFromConfigRejectsUnsafeModes`,
  `TestLoadSecretsFromConfigRejectsNonRegularAndOversizedEntries`, and
  `TestLoadSecretsFromConfigRejectsSymlinkAndAncestorReplacement`.
- Preserve: bounded reads, valid environment names, existing variables (including
  empty values) winning, regular-file checks, symlink rejection, and failure cases.
- Adapt: use ConfigKit's deterministic report/preflight/apply state machine and
  fixed budgets. Logging, configuration-driven implicit invocation, private-mode
  conventions, and application API-key plumbing remain in Narratio.

Implementation record (2026-09-06): `LoadSecretsDir`, `readSecretsEntries`,
`readSecretValue`, and the private environment/file seams in `secrets.go` adapt
Narratio's bounded-read and preserve-existing mechanisms at revision
`b68435128be1e24b94fc7ec259d6e9093669fd99`. The corresponding tests retain its
valid-file, existing-variable (including empty), missing/permission/read,
nonregular, symlink, and byte-limit cases. ConfigKit validates invalid entries
rather than skipping them, permits ordinary deployment permission modes, resolves
the selected directory symlink, performs complete preflight before mutation, and
emits per-file safe records without application configuration or logging.

## WeatherReporter evidence

### Secrets bootstrap

Assigned to the explicit bootstrap operation:

- Production: `loadSecrets`, `applySecrets`, `restoreEnvironment`, `secretValue`,
  and `environmentValue` in `internal/config/secrets.go`.
- Cases: `TestLoadSecretsDisabledLeavesEnvironmentUnchanged`,
  `TestLoadSecretsOverwritesExistingEnvironment`,
  `TestLoadSecretsTrimsOneTrailingLineEnding`,
  `TestLoadSecretsRejectsInvalidDirectoryEntries`,
  `TestLoadSecretsRejectsMissingDirectory`,
  `TestLoadFileSecretDirectoryFailureLeavesEnvironmentUnchanged`,
  `TestLoadFileValidationFailureLeavesSecretEnvironmentUnset`, and
  `TestApplySecretsRollsBackOnEnvironmentFailure`.
- Preserve: separate read/preflight and apply mechanisms, one-newline handling,
  invalid-name/type/read coverage, no writes after preflight failure, and controlled
  environment-write failure coverage.
- Adapt: existing variables—including empty ones—are skipped, never overwritten;
  application failures produce an accurate partial report and no rollback. Skip
  entries are not read. Bootstrap is explicitly invoked and remains independent of
  YAML loading and semantic validation.

Implementation record (2026-09-06): the same `secrets.go` state machine adapts
WeatherReporter's `loadSecrets`/`applySecrets` separation, environment-name check,
regular-file rejection, and exact one-LF/CRLF removal at revision
`5d306a202471e10fe98923a1df27d0b0fce5c666`. Tests adapt its newline matrix,
invalid-entry and preflight-no-write cases, and controlled application failure.
The overwrite/rollback expectation is intentionally replaced by existing-winner,
presence-recheck, and partial-progress statuses; no configuration loader invokes
the bootstrap.

### Defaults, explicit presence, and module options

Assigned to consumer-shaped examples and migration checks:

- Production: `Load`, `mergeFile`, `normalizeReportModules`,
  `traverseReportModules`, `normalizeModuleOptions`, and `decodeKnownOptions`.
- Cases: `TestDefaults`, `TestOutputDirectoryLoading`,
  `TestOutputDirectoryValidationIsConsistentForLoadedAndConstructedConfigs`,
  `TestLoadMinimalExampleConfig`,
  `TestReportDistributorPathTemplatesSetTracksExplicitEmptyList`,
  `TestValidateReportModuleKeysWithoutMutatingOptions`,
  `TestReportModuleOverridesNormalizesConstructedOptionsWithoutMutatingConfig`,
  `TestReportModuleOverridesCanonicalizesTypedPointerOptions`,
  `TestReportDistributorPathOverridesConsistentForLoadedAndConstructedConfig`,
  `TestReportModuleValidationConsistentForLoadedAndConstructedConfig`,
  `TestConstructedReportOverridesUseExportedFields`, and
  `TestConstructedExplicitEmptyReportOverridesAreRejected`.
- Preserve: defaults-only use, consistent loaded/constructed outcomes, explicit
  empty-list domain rejection, typed module-specific options, normalization without
  mutating caller input, and unknown-field rejection.
- Adapt: ConfigKit permits empty lists structurally and exposes their presence to
  application validation. Object bindings replace shorthand, explicitly set flags
  become CLI override records, and general environment overlays retire. Report
  semantics, registries, rendering, and module option validation remain application
  responsibilities.

Consumer-example implementation record (2026-09-06): at Narratio revision
`b68435128be1e24b94fc7ec259d6e9093669fd99`, the maintained consumer cases adapt
`TestMaintainedExtractionExamplesPreservePublishedContracts`,
`TestConfigValidateAndShowPipelineOnlyAreSideEffectFree`, and the stable source-
ownership tests into a sanitized partial-import/artifact fixture. ConfigKit replaces
application composition envelopes and command output with `_configkit.imports`,
typed defaults, recorded resolution, metadata, and explanation. Artifact meaning,
model/pipeline profiles, commands, and generated-domain attribution remain in the
example application layer; no production function was copied.

At WeatherReporter revision `5d306a202471e10fe98923a1df27d0b0fce5c666`,
`TestDefaults`, explicit-empty-list cases, loaded/constructed consistency cases, and
nonmutating module-option normalization were adapted into the WeatherReporter-shaped
fixtures and tests in [`examples/consumers`](../../examples/consumers). ConfigKit's
detached defaults replace source mutation, canonical module objects replace string
shorthand, ordinary validation retains empty-list and option compatibility rules,
and explicit `Override` records replace external configuration-field overlays. The
startup example composes the separately implemented bootstrap report, typed secret
lookup, and explanation APIs without copying WeatherReporter's registries or
rendering/runtime modules.

Notarius revision `77ae132e5609916da40dcd4fcc749c000a63cbac` supplied
binding/version/dynamic-option migration evidence only. The local example uses an
ordinary canonical object, a string `VersionRule`, ConfigKit's `int64` dynamic
integer convention, and application range checking; it does not copy a catalog or
environment overlay. Seriatim revision
`0fc92f364316c874d366b00cdc884a33e1900b25` supplied ordered rule-file evidence.
The local examples load speaker and autocorrect files independently and keep their
matching/replacement algorithms outside ConfigKit. Neither source is a build or test
dependency.

## Application-only material

Do not extract application commands, filesystem discovery, logging, runtime
initialization, remote clients, LLM/provider setup, audio/transcript fixtures,
configuration digests, report rendering, campaign/session algorithms, or consumer
resource validation into ConfigKit. Maintained consumer examples use only small
synthetic types and fixtures to prove that these responsibilities remain
straightforward.