84 lines
6.7 KiB
Markdown
84 lines
6.7 KiB
Markdown
# Destination State Internals
|
|
|
|
Audience: developers and LLM coding agents changing `internal/state`.
|
|
|
|
## Purpose
|
|
|
|
`internal/state` parses, validates, serializes, and compares `.distributor.json` destination state.
|
|
|
|
## Inputs And Outputs
|
|
|
|
Inputs are destination state JSON, constructed state values, current source manifest, pipeline id, destination id, and whether the destination path has content without state. Outputs are validated state values, JSON bytes, comparison outcomes, and human-readable reasons.
|
|
|
|
## Boundaries
|
|
|
|
The package does not inspect storage backends, mutate files, choose transfer policy, build publish outputs, generate URLs, or parse config. Publish planning consumes state comparison outcomes.
|
|
|
|
The external destination state contract is documented in `docs/integrations/destination-state.md`.
|
|
|
|
## Config Fields Used
|
|
|
|
`internal/state` uses config state mode and reconciliation mode constants for destination state validation and legacy state normalization. Destination ids, pipeline ids, and link URLs originate from config but are supplied as values by callers.
|
|
|
|
## Adapters Used
|
|
|
|
None.
|
|
|
|
## State And Manifest Behavior
|
|
|
|
`.distributor.json` schema version is `2` for newly written single-owner state. Required fields are `pipeline_id`, `destination_id`, `published_at`, `created_at`, `updated_at`, `state.mode`, `reconciliation.mode`, `source.manifest`, and `outputs`. `distributor_version` and `links` are optional.
|
|
|
|
Schema version `1` state remains readable. Parsing infers `state.mode: single_owner`, `reconciliation.mode: replace`, top-level `created_at` and `updated_at` from `published_at`, and per-output timestamps from `published_at`.
|
|
|
|
Schema version `3` is shared-root state. It records `state.mode: shared_root`, shared state timestamps, owner records keyed by pipeline id and destination id, each owner's latest source manifest and reconciliation metadata, optional owner primary links, and output records for every managed path. Shared-root output records include owner ids and compact source identity fields for source id, digest, and creation time.
|
|
|
|
Shared-root publish conversion is explicit. Compatible single-owner state for the same pipeline and destination can be projected into the current owner scope by publish execution. Unrelated single-owner state remains a conflict.
|
|
|
|
Embedded source manifests are parsed and validated through `internal/bundle`, which delegates source manifest semantics to `pkg/bundle`. Output records require clean paths, `source` or `generated` kind, valid source paths, lowercase SHA-256 digests, non-negative sizes, created and updated timestamps, and transform ids for generated outputs. Stored URLs must pass `internal/link` validation.
|
|
|
|
The package also provides helpers for finding output records by path, projecting planned publish outputs into timestamped state outputs, merging retained and newly planned output records, computing managed output paths from single-owner state, removing missing managed output records from single-owner state, and building prune candidates from managed outputs.
|
|
|
|
For shared-root state, helpers parse either state shape, identify the current owner scope, return an owner's latest source manifest, list managed paths for one owner or all owners, detect path ownership conflicts, project planned owner outputs, merge one owner's planned outputs while preserving unrelated owners, replace one owner's outputs by removing that owner's omitted outputs, remove missing managed output records for either the current owner or every owner, and build owner-scoped prune candidates.
|
|
|
|
Shared-root helper projections preserve output `created_at` for existing managed paths and use the current publication time for rewritten `updated_at`. Root-level `created_at` preservation is owned by publish execution.
|
|
|
|
## Skip And Resume Behavior
|
|
|
|
Comparison is pure. It returns outcomes for absent state, unmanaged content, invalid state, pipeline/destination mismatch, same source manifest, older destination, newer destination, same-created digest conflict, and different source id conflict. It does not decide whether to skip, replace, force, or fail; publish planning maps outcomes to actions.
|
|
|
|
Shared-root comparison is owner-scoped. It compares only the owner keyed by the current pipeline id and destination id, treats an absent owner as absent destination state for that owner, and can compare compatible single-owner state for the current owner without converting unrelated single-owner state.
|
|
|
|
Missing-output removal helpers are pure state transformations used by app-level state repair. They remove matching output records only and leave storage inspection, timestamp updates, validation, and state rewrites to callers.
|
|
|
|
Prune planning helpers are pure. They select managed output candidates, sort deterministically by `updated_at` and path, preserve the newest `keep_latest` candidates before evaluating `older_than`, and return planned prune/preserve lists without mutating state. App-level prune execution and the `prune` command use the missing-output removal helpers to remove only confirmed deleted records after storage deletion succeeds.
|
|
|
|
## Failure Behavior
|
|
|
|
Parsing rejects invalid JSON, trailing data, missing required fields, invalid timestamps, invalid state mode, invalid reconciliation mode, invalid embedded manifests, duplicate owners, duplicate outputs, invalid output paths, unsupported output kinds, missing generated transforms, invalid URLs, invalid digests, negative sizes, and shared-root outputs whose owner is not registered.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/state/distributor_test.go`
|
|
- `internal/state/shared_root_test.go`
|
|
- `internal/state/prune_test.go`
|
|
- `internal/state/compare_test.go`
|
|
- `internal/app/reconcile_state_test.go`
|
|
- `internal/cli/reconcile_state_test.go`
|
|
- `internal/publish/*_test.go`
|
|
|
|
## Architectural Invariants
|
|
|
|
- `.distributor.json` is the destination sentinel and state record.
|
|
- Comparison does not mutate storage.
|
|
- Embedded source manifests use the source bundle contract.
|
|
- Newly written single-owner state uses schema version `2`.
|
|
- Schema version `1` state remains readable as replacement-mode single-owner state.
|
|
- Schema version `3` shared-root state is parsed and validated without converting unrelated single-owner state.
|
|
- Shared-root owner updates preserve unrelated owners and reject planned path collisions with other owners.
|
|
- Missing-output repair helpers preserve unrelated owner records and outputs.
|
|
- Prune planning uses output `updated_at` and preserves unrelated shared-root owners.
|
|
- Generated outputs always record a transform id.
|
|
- Output records always carry created and updated timestamps after parsing.
|
|
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
|
|
- `distributor_version` is diagnostic metadata, not a comparison key.
|