Establish ConfigKit module and extraction baseline
This commit is contained in:
242
docs/internal/extraction.md
Normal file
242
docs/internal/extraction.md
Normal file
@@ -0,0 +1,242 @@
|
||||
# Source Extraction Inventory
|
||||
|
||||
This inventory records reusable implementation evidence from sibling applications.
|
||||
Those checkouts are review inputs only: ConfigKit does not import them, and its
|
||||
build and tests must not require them. ConfigKit's current contracts and roadmap
|
||||
govern every adaptation.
|
||||
|
||||
## Inspected revisions
|
||||
|
||||
| Application | Revision | Worktree | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| Narratio | `b68435128be1e24b94fc7ec259d6e9093669fd99` | Clean `main` tracking `origin/main` | Primary parsing, composition, import, profile, presence, defaults, resolution, and inspection source |
|
||||
| WeatherReporter | `5d306a202471e10fe98923a1df27d0b0fce5c666` | Clean `main` tracking `origin/main` | Secrets bootstrap and simple-consumer/defaults/options evidence |
|
||||
|
||||
These revisions were inspected on 2026-09-06. Paths below are relative to each
|
||||
application repository. Record later source revisions and adaptations here when
|
||||
the owning implementation work extracts a symbol or case.
|
||||
|
||||
## Adaptation rules
|
||||
|
||||
- Copy and generalize only the smallest applicable mechanism or behavioral case.
|
||||
- Replace upstream YAML nodes, application types, commands, logging, digests, and
|
||||
provider registries with ConfigKit-owned boundaries.
|
||||
- Preserve strictness, deterministic behavior, and useful failure coverage while
|
||||
changing expectations that conflict with ConfigKit's accepted contracts.
|
||||
- Use sanitized local fixtures only. Audio, transcripts, credentials, private
|
||||
configuration, and whole application package trees are application-only.
|
||||
- Record the exact copied/generalized symbols and tests in the owning section when
|
||||
extraction occurs. An entry below assigns evidence; it does not claim extraction
|
||||
or behavior is implemented.
|
||||
|
||||
## Narratio evidence
|
||||
|
||||
### Source parsing and strict input
|
||||
|
||||
Assigned to the YAML/source implementation:
|
||||
|
||||
- Production: `internal/config/load.go` functions `decodeStrictYAML`,
|
||||
`decodeStrictYAMLFromReader`, and `LoadSessionBytesWithOptions`; and
|
||||
`internal/config/composition.go` functions `parseCompositionDocument`,
|
||||
`parseCompositionBytes`, and `buildCompositionNode`.
|
||||
- Cases: `TestLoadersRejectTrailingYAMLDocuments`,
|
||||
`TestStrictYAMLRejectsParserExposedEmptyTrailingDocument`, and
|
||||
`TestStrictYAMLAcceptsSingleDocumentAndRejectsUnknownFields` in
|
||||
`internal/config/loader_strict_test.go`; plus duplicate-key, alias, non-mapping,
|
||||
malformed-input, scalar spelling, presence, declaration-order, and source-label
|
||||
coverage in `TestParseCompositionDocumentRetainsPresenceOwnershipAndDeclarationOrder`
|
||||
and `TestParseCompositionDocumentRejectsAmbiguousOrMalformedYAML`.
|
||||
- Preserve: labeled reader/byte input, strict unknown-field intent, rejection of
|
||||
trailing documents (including an empty second document), duplicate detection,
|
||||
and source-aware failures.
|
||||
- Adapt: translate v4 parser nodes into ConfigKit's private document tree, apply
|
||||
ConfigKit's scalar grammar and limits, and replace raw parser errors with safe
|
||||
diagnostics. Narratio's application decoders and YAML node types are not copied.
|
||||
|
||||
### Strict composition
|
||||
|
||||
Assigned to document composition:
|
||||
|
||||
- Production: `mergeAdditiveComposition`, `mergeAdditiveCompositions`,
|
||||
`validateAdditiveClaims`, `appendCompositionClaims`, `mergeAdditiveNodes`,
|
||||
`newCompositionConflict`, and cloning/path/source helpers in
|
||||
`internal/config/composition.go`.
|
||||
- Cases: `TestMergeAdditiveCompositionJoinsOnlyDisjointMappings`,
|
||||
`TestMergeAdditiveCompositionRejectsEveryDuplicateClass`, and
|
||||
`TestMergeAdditiveCompositionReportsAllClaimingSources`.
|
||||
- Preserve: recursive joining of disjoint mappings, collisions for every overlapping
|
||||
non-mapping value (including equal values and sequences), all claimant sources,
|
||||
deterministic traversal, and nonmutation of inputs.
|
||||
- Adapt: empty mappings follow ConfigKit's no-child operation rule; paths use the
|
||||
ConfigKit semantic path model; both precise locations and composite origins are
|
||||
retained. Default values never participate in strict composition.
|
||||
|
||||
### Imports
|
||||
|
||||
Assigned to explicit import traversal and glob handling:
|
||||
|
||||
- Production: `loadPipelineCompositionSources`,
|
||||
`splitPipelineCompositionEnvelope`, `loadPipelineImports`, and related ownership
|
||||
structures in `internal/config/pipeline_composition.go`.
|
||||
- Cases: `TestSplitPipelineCompositionEnvelopePreservesInputAndDeclarationOrder`,
|
||||
`TestSplitPipelineCompositionEnvelopeRejectsMalformedShapesWithoutMutation`,
|
||||
`TestLoadPipelineCompositionImportsDisjointFieldsAndTracksOwnership`,
|
||||
`TestLoadPipelineCompositionMergesDisjointKeyedEntries`,
|
||||
`TestLoadPipelineCompositionRejectsBaseConflictsWithAllSources`,
|
||||
`TestLoadPipelineCompositionRejectsUnsafeOrInvalidImports`,
|
||||
`TestLoadPipelineCompositionRejectsSameFileAliasesAndImportedComposition`,
|
||||
`TestLoadPipelineCompositionReportsImportedParseAndSchemaSources`, and
|
||||
`TestLoadPipelineCompositionKeepsRelativePathsRootBased`.
|
||||
- Preserve: explicit declarations, declaration-order traversal, disjoint
|
||||
composition, conflict attribution, malformed/unreadable target failures, directive
|
||||
removal, and imported parse/schema source labels.
|
||||
- Adapt: replace the root-only `composition` envelope with recursive
|
||||
`_configkit.imports`; add path and glob declarations, bounded depth-first graph
|
||||
traversal, canonical containment, cycle/diamond routes, and fixed source IDs.
|
||||
Canonical paths—not `os.SameFile`—define identity, so hard links remain distinct.
|
||||
|
||||
### Profiles and overlay
|
||||
|
||||
Assigned to selected-profile overlay:
|
||||
|
||||
- Production: `mergeOverlayComposition`, `mergeOverlayNodes`,
|
||||
`selectPipelineProfile`, `loadPipelineProfileOverlays`, and
|
||||
`loadedProfileOverlay`.
|
||||
- Cases: `TestLoadPipelineProfilesSelectDefaultAndExplicitOverlay`,
|
||||
`TestLoadPipelineProfileSelectionErrors`,
|
||||
`TestPipelineProfileNamesRejectControlCharacters`,
|
||||
`TestLoadPipelineProfilesValidateEveryDeclaredOverlay`,
|
||||
`TestLoadPipelineProfileOverlaySemantics`,
|
||||
`TestLoadPipelineProfileRejectsNullAndKindChanges`, and
|
||||
`TestLoadPipelineProfilePairResolvesIndependentEffectivePipelines`.
|
||||
- Preserve: recursive mapping overlay, atomic scalar/sequence replacement,
|
||||
deterministic selection failures, source ownership, and independent results.
|
||||
- Adapt: load only the selected fixed profile path; lowercase `default` means no
|
||||
overlay. ConfigKit permits null and cross-kind replacement and does not support
|
||||
declared defaults, inheritance, arbitrary profile paths, or validating every
|
||||
unselected overlay.
|
||||
|
||||
### Effective inspection and provenance
|
||||
|
||||
Assigned to layered metadata, redaction, rendering, and explanation:
|
||||
|
||||
- Production: `EffectivePipelineValues`, `EffectivePipelineSources`,
|
||||
`MarshalEffectivePipeline`, `effectivePipelineFieldPaths`,
|
||||
`effectivePipelineOwners`, and composition `semanticRecords`/
|
||||
`compactSemanticRecords`.
|
||||
- Cases: `TestCompositionCanonicalOutputsAreDeterministic`,
|
||||
`TestEffectivePipelineValuesIgnoreEquivalentSourceLayout`,
|
||||
`TestConfigValidateAndShowPipelineOnlyAreSideEffectFree`,
|
||||
`TestConfigSourcesReportsStableOwnership`,
|
||||
`TestConfigSourcesReportsCanonicalFamilyAndPublishOrigins`,
|
||||
`TestConfigSourcesPreservesProfileOwnershipThroughFamilyExpansion`, and
|
||||
`TestConfigSourcesReportsRootImportProfileAndDefaultOwnership`.
|
||||
- Preserve: deterministic effective projection, stable logical paths, source
|
||||
ownership, side-effect-free inspection, and domain-attribution evidence.
|
||||
- Adapt: project directly from ConfigKit-owned typed values without YAML round trips;
|
||||
add line/column locations, composite origins, complete operation history, detached
|
||||
records, and redaction. Narratio commands, diff/digest APIs, generated artifact
|
||||
attribution, campaign logic, and party expansion remain application-owned.
|
||||
|
||||
### Defaults, presence, and domain resolution
|
||||
|
||||
Assigned across schema, decoding, resolution, validation, and consumer examples:
|
||||
|
||||
- Production: `NormalizeConfig.UnmarshalYAML`, `outputPathWasSet`,
|
||||
`applyPipelineDefaults` and its component default helpers, `Resolve`,
|
||||
`ResolveLoadedPipelineCampaign`, `mergeCampaignSession`, `selectStableInput`, and
|
||||
application-supplied byte loading in `internal/app/config_loader.go`.
|
||||
- Cases: `TestNormalizeLoadAndValidate`,
|
||||
`TestRepresentativeLoadDefaultsAndValidation`,
|
||||
`TestMarshalEffectivePipelineRendersStablePublicConfiguration`,
|
||||
`TestPipelineCampaignRegistryStrictDecode`,
|
||||
`TestCampaignSessionMergeFillsStableInputs`,
|
||||
`TestCampaignSessionMergeSessionOverridesStableInputs`,
|
||||
`TestLoadWithSessionOptionsCarriesExplicitProfilePresence`, and the maintained
|
||||
extraction-example coverage in `TestMaintainedExtractionExamplesPreservePublishedContracts`.
|
||||
- Preserve: fresh defaults, explicit-presence-sensitive domain decisions, strict
|
||||
nested types, stable input selection, application-supplied labeled bytes, and
|
||||
validation after resolution.
|
||||
- Adapt: presence comes from metadata rather than custom YAML hooks or pointer/zero
|
||||
guesses; explicit empty and null values are retained; defaults are cloned and
|
||||
application-owned; every domain mutation is recorded. Campaign/session fallback,
|
||||
search paths, fetching, storage, and concrete default values remain in Narratio.
|
||||
|
||||
### Secrets bootstrap
|
||||
|
||||
Assigned to the explicit bootstrap operation:
|
||||
|
||||
- Production: `loadSecretsFromConfigWithDependencies`, `validateSecretDirectory`,
|
||||
`validateSecretFile`, `readAPIKeyFile`, and
|
||||
`internal/fileops/read_bounded.go` function `ReadFileBounded`.
|
||||
- Cases: `TestLoadSecretsFromConfigLoadsValidFiles`,
|
||||
`TestLoadSecretsFromConfigPreservesExistingEnv`,
|
||||
`TestLoadSecretsFromConfigRelativeDirUsesCWD`,
|
||||
`TestLoadSecretsFromConfigMissingDirFails`,
|
||||
`TestLoadSecretsFromConfigUnreadableValidEntryFails`,
|
||||
`TestLoadSecretsFromConfigRejectsUnsafeModes`,
|
||||
`TestLoadSecretsFromConfigRejectsNonRegularAndOversizedEntries`, and
|
||||
`TestLoadSecretsFromConfigRejectsSymlinkAndAncestorReplacement`.
|
||||
- Preserve: bounded reads, valid environment names, existing variables (including
|
||||
empty values) winning, regular-file checks, symlink rejection, and failure cases.
|
||||
- Adapt: use ConfigKit's deterministic report/preflight/apply state machine and
|
||||
fixed budgets. Logging, configuration-driven implicit invocation, private-mode
|
||||
conventions, and application API-key plumbing remain in Narratio.
|
||||
|
||||
## WeatherReporter evidence
|
||||
|
||||
### Secrets bootstrap
|
||||
|
||||
Assigned to the explicit bootstrap operation:
|
||||
|
||||
- Production: `loadSecrets`, `applySecrets`, `restoreEnvironment`, `secretValue`,
|
||||
and `environmentValue` in `internal/config/secrets.go`.
|
||||
- Cases: `TestLoadSecretsDisabledLeavesEnvironmentUnchanged`,
|
||||
`TestLoadSecretsOverwritesExistingEnvironment`,
|
||||
`TestLoadSecretsTrimsOneTrailingLineEnding`,
|
||||
`TestLoadSecretsRejectsInvalidDirectoryEntries`,
|
||||
`TestLoadSecretsRejectsMissingDirectory`,
|
||||
`TestLoadFileSecretDirectoryFailureLeavesEnvironmentUnchanged`,
|
||||
`TestLoadFileValidationFailureLeavesSecretEnvironmentUnset`, and
|
||||
`TestApplySecretsRollsBackOnEnvironmentFailure`.
|
||||
- Preserve: separate read/preflight and apply mechanisms, one-newline handling,
|
||||
invalid-name/type/read coverage, no writes after preflight failure, and controlled
|
||||
environment-write failure coverage.
|
||||
- Adapt: existing variables—including empty ones—are skipped, never overwritten;
|
||||
application failures produce an accurate partial report and no rollback. Skip
|
||||
entries are not read. Bootstrap is explicitly invoked and remains independent of
|
||||
YAML loading and semantic validation.
|
||||
|
||||
### Defaults, explicit presence, and module options
|
||||
|
||||
Assigned to consumer-shaped examples and migration checks:
|
||||
|
||||
- Production: `Load`, `mergeFile`, `normalizeReportModules`,
|
||||
`traverseReportModules`, `normalizeModuleOptions`, and `decodeKnownOptions`.
|
||||
- Cases: `TestDefaults`, `TestOutputDirectoryLoading`,
|
||||
`TestOutputDirectoryValidationIsConsistentForLoadedAndConstructedConfigs`,
|
||||
`TestLoadMinimalExampleConfig`,
|
||||
`TestReportDistributorPathTemplatesSetTracksExplicitEmptyList`,
|
||||
`TestValidateReportModuleKeysWithoutMutatingOptions`,
|
||||
`TestReportModuleOverridesNormalizesConstructedOptionsWithoutMutatingConfig`,
|
||||
`TestReportModuleOverridesCanonicalizesTypedPointerOptions`,
|
||||
`TestReportDistributorPathOverridesConsistentForLoadedAndConstructedConfig`,
|
||||
`TestReportModuleValidationConsistentForLoadedAndConstructedConfig`,
|
||||
`TestConstructedReportOverridesUseExportedFields`, and
|
||||
`TestConstructedExplicitEmptyReportOverridesAreRejected`.
|
||||
- Preserve: defaults-only use, consistent loaded/constructed outcomes, explicit
|
||||
empty-list domain rejection, typed module-specific options, normalization without
|
||||
mutating caller input, and unknown-field rejection.
|
||||
- Adapt: ConfigKit permits empty lists structurally and exposes their presence to
|
||||
application validation. Object bindings replace shorthand, explicitly set flags
|
||||
become CLI override records, and general environment overlays retire. Report
|
||||
semantics, registries, rendering, and module option validation remain application
|
||||
responsibilities.
|
||||
|
||||
## Application-only material
|
||||
|
||||
Do not extract application commands, filesystem discovery, logging, runtime
|
||||
initialization, remote clients, LLM/provider setup, audio/transcript fixtures,
|
||||
configuration digests, report rendering, campaign/session algorithms, or consumer
|
||||
resource validation into ConfigKit. Later consumer examples may use small synthetic
|
||||
types and fixtures to prove that these responsibilities remain straightforward.
|
||||
Reference in New Issue
Block a user