ConfigKit/docs/internal/extraction.md

Source Extraction Inventory

This inventory records reusable implementation evidence from sibling applications. Those checkouts are review inputs only: ConfigKit does not import them, and its build and tests must not require them. ConfigKit's current contracts and roadmap govern every adaptation.

Inspected revisions

Application	Revision	Worktree	Role
Narratio	b68435128be1e24b94fc7ec259d6e9093669fd99	Clean main tracking origin/main	Primary parsing, composition, import, profile, presence, defaults, resolution, and inspection source
WeatherReporter	5d306a202471e10fe98923a1df27d0b0fce5c666	Clean main tracking origin/main	Secrets bootstrap and simple-consumer/defaults/options evidence

These revisions were inspected on 2026-09-06. Paths below are relative to each application repository. Record later source revisions and adaptations here when the owning implementation work extracts a symbol or case.

Adaptation rules

• Copy and generalize only the smallest applicable mechanism or behavioral case.
• Replace upstream YAML nodes, application types, commands, logging, digests, and provider registries with ConfigKit-owned boundaries.
• Preserve strictness, deterministic behavior, and useful failure coverage while changing expectations that conflict with ConfigKit's accepted contracts.
• Use sanitized local fixtures only. Audio, transcripts, credentials, private configuration, and whole application package trees are application-only.
• Record the exact copied/generalized symbols and tests in the owning section when extraction occurs. An entry below assigns evidence; it does not claim extraction or behavior is implemented.

Narratio evidence

Source parsing and strict input

Assigned to the YAML/source implementation:

• Production: internal/config/load.go functions decodeStrictYAML, decodeStrictYAMLFromReader, and LoadSessionBytesWithOptions; and internal/config/composition.go functions parseCompositionDocument, parseCompositionBytes, and buildCompositionNode.
• Cases: TestLoadersRejectTrailingYAMLDocuments, TestStrictYAMLRejectsParserExposedEmptyTrailingDocument, and TestStrictYAMLAcceptsSingleDocumentAndRejectsUnknownFields in internal/config/loader_strict_test.go; plus duplicate-key, alias, non-mapping, malformed-input, scalar spelling, presence, declaration-order, and source-label coverage in TestParseCompositionDocumentRetainsPresenceOwnershipAndDeclarationOrder and TestParseCompositionDocumentRejectsAmbiguousOrMalformedYAML.
• Preserve: labeled reader/byte input, strict unknown-field intent, rejection of trailing documents (including an empty second document), duplicate detection, and source-aware failures.
• Adapt: translate v4 parser nodes into ConfigKit's private document tree, apply ConfigKit's scalar grammar and limits, and replace raw parser errors with safe diagnostics. Narratio's application decoders and YAML node types are not copied.

Implementation record (2026-09-06): the labeled byte/reader ownership and strict single-document cases above were generalized into source.go, parse.go, and internal/yaml. No production line was copied verbatim. ConfigKit adds byte and tree ceilings, shared atomic reader claims, regular-file checks, exact numeric lexemes, parser-depth calibration, ConfigKit-owned locations, safe diagnostic codes, and an AST import-boundary regression. Application typed decoding, parser errors, YAML node types, and session-specific loaders were not retained.

Strict composition

Assigned to document composition:

• Production: mergeAdditiveComposition, mergeAdditiveCompositions, validateAdditiveClaims, appendCompositionClaims, mergeAdditiveNodes, newCompositionConflict, and cloning/path/source helpers in internal/config/composition.go.
• Cases: TestMergeAdditiveCompositionJoinsOnlyDisjointMappings, TestMergeAdditiveCompositionRejectsEveryDuplicateClass, and TestMergeAdditiveCompositionReportsAllClaimingSources.
• Preserve: recursive joining of disjoint mappings, collisions for every overlapping non-mapping value (including equal values and sequences), all claimant sources, deterministic traversal, and nonmutation of inputs.
• Adapt: empty mappings follow ConfigKit's no-child operation rule; paths use the ConfigKit semantic path model; both precise locations and composite origins are retained. Default values never participate in strict composition.

Implementation record (2026-09-06): composeDocuments, its pre-order claim collection, conflict construction, and recursive mapping join in compose.go adapt the assigned production symbols. TestComposeDocumentsJoinsDisjointMappingsAndTracksClaims, TestComposeDocumentsRejectsEveryPresentNonMappingOverlap, and TestComposeDocumentsReportsAllClaimantsAtExactLocations adapt the assigned cases. The planned empty-mapping difference is covered separately: empty mappings retain container provenance but contribute no child claims. No unplanned semantic deviation from the extracted additive composer was required.

Imports

Assigned to explicit import traversal and glob handling:

• Production: loadPipelineCompositionSources, splitPipelineCompositionEnvelope, loadPipelineImports, and related ownership structures in internal/config/pipeline_composition.go.
• Cases: TestSplitPipelineCompositionEnvelopePreservesInputAndDeclarationOrder, TestSplitPipelineCompositionEnvelopeRejectsMalformedShapesWithoutMutation, TestLoadPipelineCompositionImportsDisjointFieldsAndTracksOwnership, TestLoadPipelineCompositionMergesDisjointKeyedEntries, TestLoadPipelineCompositionRejectsBaseConflictsWithAllSources, TestLoadPipelineCompositionRejectsUnsafeOrInvalidImports, TestLoadPipelineCompositionRejectsSameFileAliasesAndImportedComposition, TestLoadPipelineCompositionReportsImportedParseAndSchemaSources, and TestLoadPipelineCompositionKeepsRelativePathsRootBased.
• Preserve: explicit declarations, declaration-order traversal, disjoint composition, conflict attribution, malformed/unreadable target failures, directive removal, and imported parse/schema source labels.
• Adapt: replace the root-only composition envelope with recursive _configkit.imports; add path and glob declarations, bounded depth-first graph traversal, canonical containment, cycle/diamond routes, and fixed source IDs. Canonical paths—not os.SameFile—define identity, so hard links remain distinct.

Path-import implementation record (2026-09-06): splitImportDirectives, parseImportDeclaration, importFilesystem, importTraversal.visitPath, and loadImportedDocuments in imports.go adapt the assigned envelope, loading, source, and failure mechanics. The corresponding ConfigKit tests adapt the assigned declaration-order, disjoint composition, keyed-entry, all-source conflict, unsafe target, alias, imported parse/schema, and relative-resolution cases. ConfigKit adds recursive bounded DFS, os.OpenRoot confinement, canonical symlink identity, cycle/diamond route locations, optional-path semantics, aggregate budgets, and fragment version-placement enforcement. The planned glob syntax is parsed but remains explicitly unavailable for its next owning work; hard links intentionally replace Narratio's same-file rejection with path identity.

Glob-import implementation record (2026-09-06): ConfigKit retains the extracted declaration order, import failures, composition, and identity cases while adding the new expandImportGlob, readGlobMatches, and pattern-validation mechanisms in glob.go. Focused cases cover lexical ordering, multiple declarations, overlap, directories, hard links, contained and escaping symlinks, unmatched declarations, malformed and recursive patterns, escaped metacharacters, character classes, permission failures, explicit conf.d use, and entry/candidate budgets. This work has no Narratio production equivalent; it uses the planned standard-library path.Match and bounded File.ReadDir adaptation rather than its flat loader.

Profiles and overlay

Assigned to selected-profile overlay:

• Production: mergeOverlayComposition, mergeOverlayNodes, selectPipelineProfile, loadPipelineProfileOverlays, and loadedProfileOverlay.
• Cases: TestLoadPipelineProfilesSelectDefaultAndExplicitOverlay, TestLoadPipelineProfileSelectionErrors, TestPipelineProfileNamesRejectControlCharacters, TestLoadPipelineProfilesValidateEveryDeclaredOverlay, TestLoadPipelineProfileOverlaySemantics, TestLoadPipelineProfileRejectsNullAndKindChanges, and TestLoadPipelineProfilePairResolvesIndependentEffectivePipelines.
• Preserve: recursive mapping overlay, atomic scalar/sequence replacement, deterministic selection failures, source ownership, and independent results.
• Adapt: load only the selected fixed profile path; lowercase default means no overlay. ConfigKit permits null and cross-kind replacement and does not support declared defaults, inheritance, arbitrary profile paths, or validating every unselected overlay.

Implementation record (2026-09-06): overlayDocuments, mergeOverlayMappings, loadProfileDocument, and profileDirectiveDiagnostics adapt the assigned merge, selection, loading, and ownership mechanics. TestOverlayDocumentsRecursesMappingsAndReplacesAtomicSubtrees, TestLoadAppliesSelectedProfileAfterBaseAndBeforeDefaults, profile selection/error, directive/version, containment, and budget cases adapt the assigned tests. ConfigKit uses exactly one fixed selected .yml source, permits null and cross-kind replacement, retains displaced descendant history, rejects any profile directive or version-path contribution, and counts the profile in aggregate budgets. Narratio's declared default, multi-overlay validation, inheritance-like application conventions, and null/kind restrictions were not retained.

Effective inspection and provenance

Assigned to layered metadata, redaction, rendering, and explanation:

• Production: EffectivePipelineValues, EffectivePipelineSources, MarshalEffectivePipeline, effectivePipelineFieldPaths, effectivePipelineOwners, and composition semanticRecords/ compactSemanticRecords.
• Cases: TestCompositionCanonicalOutputsAreDeterministic, TestEffectivePipelineValuesIgnoreEquivalentSourceLayout, TestConfigValidateAndShowPipelineOnlyAreSideEffectFree, TestConfigSourcesReportsStableOwnership, TestConfigSourcesReportsCanonicalFamilyAndPublishOrigins, TestConfigSourcesPreservesProfileOwnershipThroughFamilyExpansion, and TestConfigSourcesReportsRootImportProfileAndDefaultOwnership.
• Preserve: deterministic effective projection, stable logical paths, source ownership, side-effect-free inspection, and domain-attribution evidence.
• Adapt: project directly from ConfigKit-owned typed values without YAML round trips; add line/column locations, composite origins, complete operation history, detached records, and redaction. Narratio commands, diff/digest APIs, generated artifact attribution, campaign logic, and party expansion remain application-owned.

Layered-provenance implementation record (2026-09-06): ConfigKit integrated its previously extracted source claims, overlay displacement records, typed baseline, and audited resolution snapshots in load_metadata.go. The representative TestLayeredProvenanceAgreesWithResolvedSnapshot adapts Narratio's stable ownership, profile ownership, deterministic effective projection, and side-effect-free inspection cases across one default/import/profile/resolution workflow. ConfigKit adds chronological descendant history for parent and whole-sequence operations, composite empty-mapping origins, canonical resolution reasons/dependencies, private snapshot agreement, defensive nested copies, and concurrent load isolation. Value rendering, redaction, explanation, commands, generated attribution, diffs, and digests remain assigned to their later owning work.

Defaults, presence, and domain resolution

Assigned across schema, decoding, resolution, validation, and consumer examples:

• Production: NormalizeConfig.UnmarshalYAML, outputPathWasSet, applyPipelineDefaults and its component default helpers, Resolve, ResolveLoadedPipelineCampaign, mergeCampaignSession, selectStableInput, and application-supplied byte loading in internal/app/config_loader.go.
• Cases: TestNormalizeLoadAndValidate, TestRepresentativeLoadDefaultsAndValidation, TestMarshalEffectivePipelineRendersStablePublicConfiguration, TestPipelineCampaignRegistryStrictDecode, TestCampaignSessionMergeFillsStableInputs, TestCampaignSessionMergeSessionOverridesStableInputs, TestLoadWithSessionOptionsCarriesExplicitProfilePresence, and the maintained extraction-example coverage in TestMaintainedExtractionExamplesPreservePublishedContracts.
• Preserve: fresh defaults, explicit-presence-sensitive domain decisions, strict nested types, stable input selection, application-supplied labeled bytes, and validation after resolution.
• Adapt: presence comes from metadata rather than custom YAML hooks or pointer/zero guesses; explicit empty and null values are retained; defaults are cloned and application-owned; every domain mutation is recorded. Campaign/session fallback, search paths, fetching, storage, and concrete default values remain in Narratio.

Implementation record (2026-09-06): ConfigKit generalized Narratio's fresh-default, strict nested-type, and explicit-presence cases into decode.go and decode_test.go. The decoder applies the owned document tree over a detached typed baseline, treats mappings recursively and sequences atomically, preserves explicit false/zero/empty/null, rejects unknown nested fields, and returns private presence and default/input origin artifacts. Unlike Narratio's custom YAML hooks and pointer/zero inference, presence comes directly from the parsed tree and equal-valued input remains explicit. ConfigKit also adds destination-width numeric checks, dynamic int64/float64, exact arrays, duration strings, paired text codecs, safe multi-error collection, and repeated-result isolation. Resolution, validation, consumer defaults, and all campaign/session algorithms remain assigned to later work.

Lifecycle integration record (2026-09-06): the public single-source Load boundary combines the already extracted labeled-source and strict-decoding mechanisms with Narratio's fresh-default and presence expectations. ConfigKit adds an application- defined pre-decode string-version check, defaults-only NoSource operation, detached base/default metadata and source reports, and explicit gates for every later lifecycle option. No application loader or production line was copied. Narratio's resolution, validation, import/profile, campaign/session, and command behavior remains assigned to its later owning work.

Resolution implementation record (2026-09-06): ConfigKit adapted the intent of Narratio's Resolve, applyPipelineDefaults, component default helpers, mergeCampaignSession, and selectStableInput into a generic recorded resolver in resolution.go. The synthetic timeout/artifact cases preserve explicit-input decisions, derived dependencies, normalization, and map copy/writeback without copying campaign/session algorithms or concrete defaults. ConfigKit adds structural mutation auditing, container coverage, helper/path consistency checks, equal-value intent, coalesced repeated writes, private before/after snapshots, deterministic record ordering, and safe hook diagnostics. Semantic validation remains assigned to the next lifecycle work.

Validation implementation record (2026-09-06): ConfigKit generalized Narratio's TestNormalizeLoadAndValidate, TestRepresentativeLoadDefaultsAndValidation, strict registry decoding, and maintained extraction-example expectations into the post-resolution Validation lifecycle and synthetic consumer examples. ConfigKit keeps domain rules application-owned, resolves schema-valid missing/dynamic report paths to nearest input locations, collects independent warnings/errors, and audits the full typed projection to reject validator mutation. Live resources, registries, campaign/session logic, commands, and application runtime initialization were not extracted.

Secrets bootstrap

Assigned to the explicit bootstrap operation:

• Production: loadSecretsFromConfigWithDependencies, validateSecretDirectory, validateSecretFile, readAPIKeyFile, and internal/fileops/read_bounded.go function ReadFileBounded.
• Cases: TestLoadSecretsFromConfigLoadsValidFiles, TestLoadSecretsFromConfigPreservesExistingEnv, TestLoadSecretsFromConfigRelativeDirUsesCWD, TestLoadSecretsFromConfigMissingDirFails, TestLoadSecretsFromConfigUnreadableValidEntryFails, TestLoadSecretsFromConfigRejectsUnsafeModes, TestLoadSecretsFromConfigRejectsNonRegularAndOversizedEntries, and TestLoadSecretsFromConfigRejectsSymlinkAndAncestorReplacement.
• Preserve: bounded reads, valid environment names, existing variables (including empty values) winning, regular-file checks, symlink rejection, and failure cases.
• Adapt: use ConfigKit's deterministic report/preflight/apply state machine and fixed budgets. Logging, configuration-driven implicit invocation, private-mode conventions, and application API-key plumbing remain in Narratio.

WeatherReporter evidence

Secrets bootstrap

Assigned to the explicit bootstrap operation:

• Production: loadSecrets, applySecrets, restoreEnvironment, secretValue, and environmentValue in internal/config/secrets.go.
• Cases: TestLoadSecretsDisabledLeavesEnvironmentUnchanged, TestLoadSecretsOverwritesExistingEnvironment, TestLoadSecretsTrimsOneTrailingLineEnding, TestLoadSecretsRejectsInvalidDirectoryEntries, TestLoadSecretsRejectsMissingDirectory, TestLoadFileSecretDirectoryFailureLeavesEnvironmentUnchanged, TestLoadFileValidationFailureLeavesSecretEnvironmentUnset, and TestApplySecretsRollsBackOnEnvironmentFailure.
• Preserve: separate read/preflight and apply mechanisms, one-newline handling, invalid-name/type/read coverage, no writes after preflight failure, and controlled environment-write failure coverage.
• Adapt: existing variables—including empty ones—are skipped, never overwritten; application failures produce an accurate partial report and no rollback. Skip entries are not read. Bootstrap is explicitly invoked and remains independent of YAML loading and semantic validation.

Defaults, explicit presence, and module options

Assigned to consumer-shaped examples and migration checks:

• Production: Load, mergeFile, normalizeReportModules, traverseReportModules, normalizeModuleOptions, and decodeKnownOptions.
• Cases: TestDefaults, TestOutputDirectoryLoading, TestOutputDirectoryValidationIsConsistentForLoadedAndConstructedConfigs, TestLoadMinimalExampleConfig, TestReportDistributorPathTemplatesSetTracksExplicitEmptyList, TestValidateReportModuleKeysWithoutMutatingOptions, TestReportModuleOverridesNormalizesConstructedOptionsWithoutMutatingConfig, TestReportModuleOverridesCanonicalizesTypedPointerOptions, TestReportDistributorPathOverridesConsistentForLoadedAndConstructedConfig, TestReportModuleValidationConsistentForLoadedAndConstructedConfig, TestConstructedReportOverridesUseExportedFields, and TestConstructedExplicitEmptyReportOverridesAreRejected.
• Preserve: defaults-only use, consistent loaded/constructed outcomes, explicit empty-list domain rejection, typed module-specific options, normalization without mutating caller input, and unknown-field rejection.
• Adapt: ConfigKit permits empty lists structurally and exposes their presence to application validation. Object bindings replace shorthand, explicitly set flags become CLI override records, and general environment overlays retire. Report semantics, registries, rendering, and module option validation remain application responsibilities.

Application-only material

Do not extract application commands, filesystem discovery, logging, runtime initialization, remote clients, LLM/provider setup, audio/transcript fixtures, configuration digests, report rendering, campaign/session algorithms, or consumer resource validation into ConfigKit. Later consumer examples may use small synthetic types and fixtures to prove that these responsibilities remain straightforward.