Update the staged implementation plan to address remaining items from the destination policy refactor
This commit is contained in:
@@ -1,345 +1,399 @@
|
|||||||
# Catalog State And Workflow Implementation Roadmap
|
# Catalog Follow-Up Implementation Roadmap
|
||||||
|
|
||||||
This is the active staged implementation plan for
|
This is the active staged plan for closing the remaining catalog-state
|
||||||
`docs/roadmap/catalog.md`. The feature roadmap defines the target state model
|
implementation gaps found after the schema version `4` catalog workflow landed.
|
||||||
and policy decisions; this document defines the implementation sequence for an
|
It supersedes the earlier catalog implementation plan.
|
||||||
LLM coding agent to follow stage by stage.
|
|
||||||
|
|
||||||
Future behavior must remain under `docs/roadmap/` until implemented. Update
|
The current code already supports destination `workflow: additive` and
|
||||||
current-behavior docs only in the stage that implements the corresponding
|
`workflow: replacement`, writes catalog state schema version `4`, rejects legacy
|
||||||
behavior.
|
destination policy YAML fields, and routes publish execution through catalog
|
||||||
|
planning for normal runs. The remaining work is to make the implemented behavior
|
||||||
|
release-ready: restore a meaningful catalog `--force` path, synchronize current
|
||||||
|
docs and examples, and remove legacy single-owner/shared-root implementation
|
||||||
|
debt.
|
||||||
|
|
||||||
## Current Baseline
|
## Current Baseline
|
||||||
|
|
||||||
`distributor` currently writes destination `.distributor.json` using
|
- Destination config accepts `workflow: additive` and `workflow: replacement`.
|
||||||
single-owner schema version `2` or shared-root schema version `3`. Destination
|
- Omitted `workflow` defaults to `additive`.
|
||||||
behavior is selected through several low-level knobs: `state.mode`,
|
- Destination YAML fields `state`, `reconciliation`, `takeover`, and `transfer`
|
||||||
`reconciliation.mode`, `takeover.mode`, and `transfer`.
|
are rejected by strict YAML decoding.
|
||||||
|
- Successful publish writes `.distributor.json` schema version `4` with
|
||||||
The target behavior is a clean alpha break:
|
`state.mode: catalog`.
|
||||||
|
- Schema versions `1`, `2`, and `3` are treated as superseded legacy state for
|
||||||
- all newly written destination state uses schema version `4`;
|
publish planning.
|
||||||
- all destinations use `state.mode: catalog` internally;
|
- `run --force` remains in the CLI and docs, but catalog publish planning does
|
||||||
- user-facing destination behavior is selected by `workflow: additive` or
|
not currently select `force_replace`.
|
||||||
`workflow: replacement`;
|
- Current docs and examples still describe legacy config/state behavior in
|
||||||
- `workflow` defaults to `additive`;
|
several places.
|
||||||
- legacy destination policy fields are rejected, not aliased;
|
- Legacy single-owner and shared-root state types, tests, and execution helpers
|
||||||
- legacy state schema versions `1`, `2`, and `3` are not migrated.
|
remain in the codebase even though normal publish planning no longer uses
|
||||||
|
them.
|
||||||
|
|
||||||
## Implementation Principles
|
## Implementation Principles
|
||||||
|
|
||||||
- Keep the source manifest contract unchanged.
|
- Preserve the catalog state shape defined in `docs/roadmap/catalog.md`.
|
||||||
- Keep backend adapters unaware of catalog and workflow policy.
|
- Keep `workflow` as runtime config policy; do not persist it in
|
||||||
- Treat `workflow` as runtime config, not persisted state.
|
`.distributor.json`.
|
||||||
- Keep unmanaged content protected after catalog state exists.
|
- Keep unmanaged content protected by default.
|
||||||
- Preserve output `created_at` when a path changes owner; update only
|
- Keep `--force` explicit, per-run, dry-runnable, and bounded to the configured
|
||||||
`updated_at`.
|
destination bundle path.
|
||||||
- Prefer removing legacy state/config paths over compatibility shims.
|
- Do not reintroduce legacy config aliases or compatibility migration.
|
||||||
- Preserve current public APIs outside destination state/config unless the
|
- Keep backend adapters unaware of catalog, workflow, and force policy.
|
||||||
catalog roadmap explicitly changes them.
|
- Update current-behavior docs in the same stage that makes the described
|
||||||
|
behavior true.
|
||||||
|
|
||||||
## Active Implementation Stages
|
## Active Implementation Stages
|
||||||
|
|
||||||
## Stage 1: Destination Workflow Config Clean Break
|
## Stage 1: Catalog Force Planning And Execution
|
||||||
|
|
||||||
Goal: replace low-level destination policy config with the user-facing workflow
|
Goal: make `run --force` meaningful for catalog workflows while keeping normal
|
||||||
switch.
|
catalog safety conservative.
|
||||||
|
|
||||||
Implementation scope:
|
Implementation scope:
|
||||||
|
|
||||||
- Add destination `workflow` config with accepted values `additive` and
|
- Update `internal/publish` planning so `req.Force` can select
|
||||||
`replacement`.
|
`force_replace` for exceptional destructive catalog cases.
|
||||||
- Default omitted `workflow` to `additive`.
|
- Force should apply to these cases:
|
||||||
- Remove or make unsupported the destination-level config fields `state`,
|
- no valid `.distributor.json` and destination bundle path has unmanaged
|
||||||
`reconciliation`, `takeover`, and `transfer`.
|
content;
|
||||||
- Ensure configs containing those legacy fields fail clearly. Prefer strict YAML
|
- planned path collision with storage content not recorded in valid catalog
|
||||||
unknown-field failure by removing struct fields; add explicit validation only
|
state;
|
||||||
if clearer errors are needed without weakening strict decoding.
|
- invalid destination state JSON or invalid destination state fields;
|
||||||
- Preserve backend, publish, transform, path mapping, links, retention, and
|
- unsupported future destination state schema, if the operator explicitly
|
||||||
other non-policy destination fields.
|
chooses force replacement after dry-run review.
|
||||||
- Update example configs only when the implementation stage also updates
|
- Force should not be needed for ordinary valid catalog-managed upserts or
|
||||||
current docs; otherwise keep this stage focused on config code and tests.
|
replacements. Additive and replacement workflow behavior remains normal managed
|
||||||
|
behavior.
|
||||||
|
- `force_replace` must delete only the bounded destination bundle path through
|
||||||
|
the storage abstraction, then write planned outputs and schema version `4`
|
||||||
|
catalog state.
|
||||||
|
- For fixed-path destinations, the bounded destination bundle path is the
|
||||||
|
configured backend root. Dry-run output must make that clear.
|
||||||
|
- Preserve default behavior without `--force`:
|
||||||
|
- unmanaged planned path collisions fail as `fail_unmanaged`;
|
||||||
|
- invalid or future state fails as `fail_conflict`;
|
||||||
|
- no-state non-empty destinations fail as unmanaged.
|
||||||
|
- Ensure `force_replace` state creation uses the same catalog output projection
|
||||||
|
as normal publish planning.
|
||||||
|
- Keep `Force` out of config. There must be no persistent force default.
|
||||||
|
- Remove or update skipped force tests that were disabled during catalog
|
||||||
|
implementation.
|
||||||
|
|
||||||
Tests:
|
Current-behavior documentation updates:
|
||||||
|
|
||||||
- `go test ./internal/config`
|
- Update only the force-related sections of:
|
||||||
- Omitted workflow defaults to `additive`.
|
- `docs/cli.md`;
|
||||||
- `workflow: additive` and `workflow: replacement` validate.
|
- `docs/operations.md`;
|
||||||
- Unknown workflow values fail.
|
- `docs/troubleshooting.md`;
|
||||||
- Legacy `state`, `reconciliation`, `takeover`, and `transfer` fields fail.
|
- `docs/internal/publish.md`;
|
||||||
- Existing valid examples are updated or tests are adjusted in the same change if
|
- backend integration docs where they describe forced deletion boundaries.
|
||||||
examples currently use legacy fields.
|
- Document force as an exceptional catalog recovery/replacement workflow, not as
|
||||||
|
a normal way to handle valid managed catalog state.
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- New configs express destination replacement/additive intent with one field.
|
|
||||||
- No normal config path accepts legacy destination policy knobs.
|
|
||||||
|
|
||||||
## Stage 2: Catalog State Schema Version 4
|
|
||||||
|
|
||||||
Goal: implement the schema version `4` catalog state model in `internal/state`.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Add catalog state types for top-level schema version `4`, `state.mode:
|
|
||||||
catalog`, and `outputs`.
|
|
||||||
- Add compact per-output source identity with `id`, `digest`, and `created`.
|
|
||||||
- Add catalog output records with required `path`, `pipeline_id`,
|
|
||||||
`destination_id`, `source`, `kind`, `sha256`, `size`, `created_at`, and
|
|
||||||
`updated_at`.
|
|
||||||
- Add optional `source_path`, `transform`, and `url`, present only where allowed
|
|
||||||
by the catalog roadmap.
|
|
||||||
- Validate duplicate paths, invalid owner ids, invalid source identities,
|
|
||||||
invalid output paths, invalid digests, negative sizes, invalid timestamps,
|
|
||||||
invalid kind/transform combinations, and invalid URLs.
|
|
||||||
- Marshal catalog state deterministically with the existing JSON formatting
|
|
||||||
conventions.
|
|
||||||
- Update `ParseDocument` so schema version `4` returns catalog state.
|
|
||||||
- Treat schema versions lower than `4` as superseded legacy state for publish
|
|
||||||
planning, not as readable/migrated active state. Keep enough detection to
|
|
||||||
identify legacy state and avoid treating it as arbitrary invalid JSON.
|
|
||||||
- Reject unsupported future schema versions.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/state`
|
|
||||||
- Parse/validate/marshal valid catalog state.
|
|
||||||
- Reject malformed catalog state and invalid output records.
|
|
||||||
- Detect schema versions `1`, `2`, and `3` as superseded legacy state.
|
|
||||||
- Reject future schema versions.
|
|
||||||
- Prove no top-level `owners`, `sources`, workflow, source manifest, pipeline id,
|
|
||||||
destination id, or published timestamp is accepted for catalog state.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- State package has one canonical schema version `4` catalog contract for new
|
|
||||||
writes.
|
|
||||||
- Legacy state is detected but not migrated.
|
|
||||||
|
|
||||||
## Stage 3: Catalog Publish Planning
|
|
||||||
|
|
||||||
Goal: plan publish actions against catalog state and destination workflow.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Replace publish request inputs that consume state/reconciliation/takeover and
|
|
||||||
transfer policy with destination workflow.
|
|
||||||
- Map workflow to catalog actions:
|
|
||||||
- `workflow: additive`: upsert planned outputs and retain all other managed
|
|
||||||
catalog outputs.
|
|
||||||
- `workflow: replacement`: upsert planned outputs and delete catalog outputs
|
|
||||||
owned by the current pipeline/destination that are omitted from the plan.
|
|
||||||
- Planned paths that already exist as catalog-managed outputs may be overwritten
|
|
||||||
and become owned by the current pipeline/destination.
|
|
||||||
- Planned path collisions with storage content not recorded in catalog state
|
|
||||||
fail as unmanaged content.
|
|
||||||
- Matching planned outputs may skip writes when source identity and output digest
|
|
||||||
already match, if that optimization can be implemented without changing
|
|
||||||
externally visible results; otherwise writing idempotently is acceptable.
|
|
||||||
- Existing schema `< 4` state is superseded:
|
|
||||||
- replacement workflow may plan a bounded destination-root clear before
|
|
||||||
writing planned outputs;
|
|
||||||
- additive workflow may plan overwrites for planned paths only and leave
|
|
||||||
unplanned files unmanaged.
|
|
||||||
- Invalid JSON or future schema state remains a conflict, not a superseded
|
|
||||||
legacy state.
|
|
||||||
- Preserve path mapping, publish policy, transforms, links, fixed-path
|
|
||||||
selection, and output collision checks.
|
|
||||||
|
|
||||||
Tests:
|
Tests:
|
||||||
|
|
||||||
- `go test ./internal/publish`
|
- `go test ./internal/publish`
|
||||||
- Additive workflow publishes new catalog state.
|
|
||||||
- Additive workflow overwrites an existing managed output and retains unrelated
|
|
||||||
outputs.
|
|
||||||
- Replacement workflow removes omitted outputs owned by the current
|
|
||||||
pipeline/destination.
|
|
||||||
- Replacement workflow preserves unrelated catalog outputs owned by other
|
|
||||||
pipeline/destination pairs.
|
|
||||||
- Managed output ownership moves to the current pipeline/destination when a
|
|
||||||
planned path is overwritten.
|
|
||||||
- `created_at` is preserved when an existing path changes owner; `updated_at`
|
|
||||||
changes.
|
|
||||||
- Unmanaged path collisions fail.
|
|
||||||
- Legacy schema `< 4` state follows the superseded-state rules above.
|
|
||||||
- Invalid/future state fails.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- Publish planning no longer depends on single-owner/shared-root comparison
|
|
||||||
semantics.
|
|
||||||
- Workflow behavior is fully driven by `workflow`.
|
|
||||||
|
|
||||||
## Stage 4: Catalog Publish Execution
|
|
||||||
|
|
||||||
Goal: execute catalog publish plans and write schema version `4` state.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Write only schema version `4` catalog `.distributor.json`.
|
|
||||||
- Implement additive execution as managed upsert of planned outputs plus catalog
|
|
||||||
state update.
|
|
||||||
- Implement replacement execution as managed upsert plus deletion of omitted
|
|
||||||
current-owner catalog outputs.
|
|
||||||
- For replacement over superseded legacy state, clear the bounded destination
|
|
||||||
root before writing planned outputs and catalog state.
|
|
||||||
- For additive over superseded legacy state, overwrite planned paths and write a
|
|
||||||
catalog containing only planned outputs; leave unplanned files unmanaged.
|
|
||||||
- Preserve cleanup behavior on failed writes:
|
|
||||||
- additive cleanup removes outputs newly created by the failed attempt where
|
|
||||||
practical;
|
|
||||||
- replacement cleanup follows existing managed-replacement safety where
|
|
||||||
practical;
|
|
||||||
- state is written only after selected outputs are written.
|
|
||||||
- Preserve link metadata, generated output digests, source output digests,
|
|
||||||
content type behavior, and backend-safe writes.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/publish ./internal/app`
|
|
||||||
- Additive execution writes planned outputs, overwrites managed planned paths,
|
|
||||||
retains unrelated managed outputs, and writes catalog state.
|
|
||||||
- Replacement execution deletes omitted current-owner outputs and preserves
|
|
||||||
unrelated owner outputs.
|
|
||||||
- Superseded legacy replacement clears bounded destination root only.
|
|
||||||
- Superseded legacy additive leaves unplanned files on disk but out of catalog
|
|
||||||
state.
|
|
||||||
- Failed writes do not leave misleading catalog state.
|
|
||||||
- Local, fake-backed SSH, and fake-backed S3 app paths exercise the same publish
|
|
||||||
behavior.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- Successful `run` writes only v4 catalog state.
|
|
||||||
- Destination outputs match additive/replacement workflow semantics.
|
|
||||||
|
|
||||||
## Stage 5: Run Reporting, Notifications, And CLI Surface
|
|
||||||
|
|
||||||
Goal: update user-visible run behavior to describe workflow/catalog actions
|
|
||||||
instead of legacy replacement/takeover actions.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Replace legacy action labels tied to state/reconciliation/takeover/transfer
|
|
||||||
with these stable workflow-oriented labels:
|
|
||||||
- `publish_new`
|
|
||||||
- `upsert_additive`
|
|
||||||
- `replace_catalog`
|
|
||||||
- `skip_same`
|
|
||||||
- `force_replace`
|
|
||||||
- `fail_unmanaged`
|
|
||||||
- `fail_conflict`
|
|
||||||
- Ensure text and JSON run summaries include workflow-relevant counters.
|
|
||||||
- Include `workflow` in run action records where useful.
|
|
||||||
- Update fixed-path dry-run warnings to describe additive upsert or replacement
|
|
||||||
clearly.
|
|
||||||
- Ensure notifications use the new action labels.
|
|
||||||
- Remove reporting assumptions that depend on `replace_older`,
|
|
||||||
`replace_newer`, `replace_conflict`, or `replace_takeover`.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/app ./internal/cli`
|
- `go test ./internal/app ./internal/cli`
|
||||||
- Text dry-run output distinguishes additive from replacement workflow.
|
- `go test ./internal/adapters/local ./internal/storage ./internal/storage/fake`
|
||||||
- JSON output includes workflow and stable action labels.
|
- Planning tests:
|
||||||
- Summary counters are deterministic.
|
- no-state non-empty destination returns `fail_unmanaged` without force and
|
||||||
- Notifications fire for additive and replacement writes.
|
`force_replace` with force;
|
||||||
- Existing CLI commands still parse and execute with the new config shape.
|
- unmanaged planned path collision returns `fail_unmanaged` without force and
|
||||||
|
`force_replace` with force;
|
||||||
|
- invalid state returns `fail_conflict` without force and `force_replace` with
|
||||||
|
force;
|
||||||
|
- future schema returns `fail_conflict` without force and `force_replace` with
|
||||||
|
force;
|
||||||
|
- valid catalog additive and replacement actions do not become
|
||||||
|
`force_replace`.
|
||||||
|
- Execution tests:
|
||||||
|
- `force_replace` clears only the destination bundle path;
|
||||||
|
- sibling paths outside the bundle path survive for local, fake S3, and fake
|
||||||
|
SSH-style backends;
|
||||||
|
- fixed-path force clears the configured backend root and is reported as such;
|
||||||
|
- resulting state is schema version `4` catalog state.
|
||||||
|
- CLI/app tests:
|
||||||
|
- dry-run `--force` reports `force_replace` without writing;
|
||||||
|
- normal `--force` executes and increments only the `force_replace` counter;
|
||||||
|
- JSON output includes stable `force_replace` action and summary fields.
|
||||||
|
|
||||||
Completion criteria:
|
Completion criteria:
|
||||||
|
|
||||||
- Operators can understand from dry-run output whether a destination will upsert
|
- `run --force` has observable, tested catalog behavior.
|
||||||
or replace managed catalog outputs.
|
- Force is still unnecessary for normal managed catalog replacement/upsert.
|
||||||
|
- Unmanaged/invalid/future-state replacement remains impossible without explicit
|
||||||
|
`--force`.
|
||||||
|
|
||||||
## Stage 6: Catalog Prune And Reconcile-State
|
## Stage 2: Current Documentation And Example Synchronization
|
||||||
|
|
||||||
Goal: update maintenance commands to operate on catalog state with the agreed
|
Goal: make all current-behavior docs and copyable examples match catalog schema
|
||||||
selector model.
|
version `4` and the `workflow` config model.
|
||||||
|
|
||||||
Implementation scope:
|
Implementation scope:
|
||||||
|
|
||||||
- Prune:
|
- Rewrite `docs/config.md` so destination behavior is described through:
|
||||||
- selected `--pipeline` and `--destination` prune only outputs currently owned
|
- `workflow: additive`;
|
||||||
by that pipeline/destination;
|
- `workflow: replacement`;
|
||||||
- no `--all-owners` prune mode in the initial catalog implementation;
|
- `publish`;
|
||||||
- do not add `--source-id`, `--path-prefix`, or `--kind` selectors.
|
- `transform`;
|
||||||
- Reconcile-state:
|
- `path_mapping`;
|
||||||
- selected `--pipeline` and `--destination` repair only outputs currently
|
- `links`;
|
||||||
owned by that pipeline/destination;
|
- `retention`.
|
||||||
- `--all-owners` repairs all catalog outputs;
|
- Remove current-behavior documentation for destination config fields:
|
||||||
- unmanaged reporting compares storage entries to all catalog output paths,
|
- `state`;
|
||||||
not just selected owner paths.
|
- `reconciliation`;
|
||||||
- Rewrite repaired/pruned state as schema version `4`.
|
- `takeover`;
|
||||||
- Remove legacy single-owner/shared-root maintenance branches.
|
- `transfer`.
|
||||||
|
- Rewrite `docs/integrations/destination-state.md` around schema version `4`
|
||||||
|
catalog state:
|
||||||
|
- top-level catalog fields;
|
||||||
|
- output record fields;
|
||||||
|
- source identity fields;
|
||||||
|
- generated-output-only `source_path` and `transform`;
|
||||||
|
- optional per-output `url`;
|
||||||
|
- superseded legacy schema handling for publish planning.
|
||||||
|
- Update `docs/operations.md`:
|
||||||
|
- additive workflow semantics;
|
||||||
|
- replacement workflow semantics;
|
||||||
|
- catalog `force_replace`;
|
||||||
|
- prune and reconcile-state catalog behavior;
|
||||||
|
- dry-run action labels.
|
||||||
|
- Update `docs/troubleshooting.md`:
|
||||||
|
- replace legacy conflict guidance with workflow/catalog guidance;
|
||||||
|
- describe rejected legacy config fields;
|
||||||
|
- describe force-only exceptional cases.
|
||||||
|
- Update `docs/cli.md`:
|
||||||
|
- run summary counters now use `publish_new`, `upsert_additive`,
|
||||||
|
`replace_catalog`, `skip_same`, `force_replace`, `fail_unmanaged`, and
|
||||||
|
`fail_conflict`;
|
||||||
|
- remove legacy action labels from current command reference.
|
||||||
|
- Update relevant internal docs:
|
||||||
|
- `docs/internal/app.md`;
|
||||||
|
- `docs/internal/config.md`;
|
||||||
|
- `docs/internal/publish.md`;
|
||||||
|
- `docs/internal/state.md`.
|
||||||
|
- Update policy docs where current architectural text still describes
|
||||||
|
single-owner/shared-root behavior as current:
|
||||||
|
- `docs/policy/architecture.md`;
|
||||||
|
- `docs/policy/development.md`.
|
||||||
|
- Update examples:
|
||||||
|
- remove or rewrite `examples/shared-root.yml`;
|
||||||
|
- remove or rewrite `examples/merge-reconciliation.yml`;
|
||||||
|
- ensure examples use `workflow` where workflow intent matters;
|
||||||
|
- keep examples valid, secret-free, and copyable.
|
||||||
|
- Do not document any future catalog selectors, state migration, or compatibility
|
||||||
|
aliases outside `docs/roadmap/`.
|
||||||
|
|
||||||
|
Tests and checks:
|
||||||
|
|
||||||
|
- `go test ./internal/config ./internal/cli`
|
||||||
|
- `go test ./...`
|
||||||
|
- Run the config/example checks already used by the test suite.
|
||||||
|
- Manual smoke checks:
|
||||||
|
- `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`
|
||||||
|
- `go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run`
|
||||||
|
- smoke any rewritten replacement/additive examples.
|
||||||
|
- Consistency searches:
|
||||||
|
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
|
||||||
|
- `rg -n "single_owner|shared_root|schema version \`2\`|schema version \`3\`" docs --glob '!docs/roadmap/**'`
|
||||||
|
- `rg -n "replace_older|replace_newer|replace_conflict|replace_takeover|takeover_mode" docs README.md examples --glob '!docs/roadmap/**'`
|
||||||
|
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
|
||||||
|
|
||||||
|
Completion criteria:
|
||||||
|
|
||||||
|
- Current docs describe implemented catalog behavior only.
|
||||||
|
- Copyable examples load successfully.
|
||||||
|
- No current doc tells users to configure rejected legacy fields.
|
||||||
|
- Destination state documentation is schema version `4` first.
|
||||||
|
|
||||||
|
## Stage 3: Catalog Idempotent Skip
|
||||||
|
|
||||||
|
Goal: make `skip_same` a real catalog no-op optimization instead of a stale
|
||||||
|
legacy action label.
|
||||||
|
|
||||||
|
Implementation scope:
|
||||||
|
|
||||||
|
- Implement `skip_same` when every planned output is already catalog-managed
|
||||||
|
with matching:
|
||||||
|
- pipeline id;
|
||||||
|
- destination id;
|
||||||
|
- source id;
|
||||||
|
- source digest;
|
||||||
|
- source created timestamp;
|
||||||
|
- output path;
|
||||||
|
- output kind;
|
||||||
|
- output digest;
|
||||||
|
- output size;
|
||||||
|
- generated output `source_path`;
|
||||||
|
- generated output `transform`;
|
||||||
|
- output URL metadata.
|
||||||
|
- Do not read destination bytes for this optimization. Trust valid catalog
|
||||||
|
metadata; bitrot detection remains a separate future concern.
|
||||||
|
- For `workflow: additive`, allow `skip_same` when all planned outputs match and
|
||||||
|
no planned output needs to be written. Unrelated catalog outputs are ignored
|
||||||
|
for the skip decision and remain retained.
|
||||||
|
- For `workflow: replacement`, allow `skip_same` only when all planned outputs
|
||||||
|
match and there are no current-owner catalog outputs that replacement workflow
|
||||||
|
would delete.
|
||||||
|
- Never return `skip_same` for superseded legacy state, invalid state, unmanaged
|
||||||
|
content, future state, or force replacement.
|
||||||
|
- `skip_same` must not write outputs, rewrite state, delete files, or notify.
|
||||||
|
- Count `skip_same` in text and JSON summaries.
|
||||||
|
- Keep behavior deterministic and identical across local, SSH, and S3
|
||||||
|
destinations.
|
||||||
|
|
||||||
|
Current-behavior documentation updates:
|
||||||
|
|
||||||
|
- Update `docs/cli.md`, `docs/operations.md`, and `docs/internal/publish.md` to
|
||||||
|
describe catalog `skip_same`.
|
||||||
|
- Document that `skip_same` is metadata-based and does not validate destination
|
||||||
|
bytes.
|
||||||
|
|
||||||
Tests:
|
Tests:
|
||||||
|
|
||||||
- `go test ./internal/state ./internal/app ./internal/cli`
|
- `go test ./internal/publish ./internal/app ./internal/cli`
|
||||||
- Prune selects only current-owner catalog outputs.
|
- Repeated additive publish with identical catalog metadata returns
|
||||||
- Prune deletes selected outputs and removes their catalog records.
|
`skip_same`.
|
||||||
- Reconcile-state selected owner removes only missing outputs for that owner.
|
- Repeated replacement publish with identical catalog metadata and no omitted
|
||||||
- `reconcile-state --all-owners` removes missing outputs for all owners.
|
current-owner outputs returns `skip_same`.
|
||||||
- Unmanaged reporting excludes all catalog-managed paths and reports unrecorded
|
- Changed source digest, output digest, URL, transform mode, owner, output size,
|
||||||
storage entries.
|
or generated source path prevents skip.
|
||||||
- JSON/text output remains stable and clear.
|
- Replacement workflow does not skip when it would delete omitted current-owner
|
||||||
|
outputs.
|
||||||
|
- `skip_same` does not write outputs or state and does not notify.
|
||||||
|
- `go test ./...`
|
||||||
|
|
||||||
Completion criteria:
|
Completion criteria:
|
||||||
|
|
||||||
- Maintenance commands operate only on catalog state and respect the locked
|
- Catalog action vocabulary has no stale action label.
|
||||||
selector rules.
|
- Repeated publish behavior is intentional, documented, and tested.
|
||||||
|
|
||||||
## Stage 7: Clean Break Removal And Documentation
|
## Stage 4: Legacy Publish And State Code Removal
|
||||||
|
|
||||||
Goal: remove legacy state/config behavior and document the implemented catalog
|
Goal: remove dead or near-dead single-owner/shared-root write and planning code
|
||||||
workflow model.
|
after catalog force behavior, docs, and skip semantics are settled.
|
||||||
|
|
||||||
Implementation scope:
|
Implementation scope:
|
||||||
|
|
||||||
- Remove dead code for writing schema version `2` single-owner and schema
|
- Remove legacy publish actions that are no longer planned or documented:
|
||||||
version `3` shared-root state.
|
- `replace_older`;
|
||||||
- Remove legacy config structs/constants/defaults/validation for destination
|
- `replace_newer`;
|
||||||
`state`, `reconciliation`, `takeover`, and `transfer` where no longer used.
|
- `replace_conflict`;
|
||||||
- Remove or rewrite tests that only assert legacy state/config behavior.
|
- `replace_takeover`;
|
||||||
- Update current-behavior docs:
|
- `skip_destination_newer`.
|
||||||
- `docs/config.md`
|
- Keep only catalog-era actions:
|
||||||
- `docs/cli.md`
|
- `publish_new`;
|
||||||
- `docs/operations.md`
|
- `upsert_additive`;
|
||||||
- `docs/troubleshooting.md`
|
- `replace_catalog`;
|
||||||
- `docs/integrations/destination-state.md`
|
- `skip_same`;
|
||||||
- relevant `docs/internal/` docs
|
- `force_replace`;
|
||||||
- `docs/policy/architecture.md`
|
- `fail_unmanaged`;
|
||||||
- `docs/policy/development.md`
|
- `fail_conflict`.
|
||||||
- Update examples to use `workflow` and remove legacy fields.
|
- Remove legacy fields from `publish.Plan`:
|
||||||
- Keep roadmap-only material out of current docs.
|
- `StateMode`;
|
||||||
- Once implemented and documented, remove or rewrite
|
- `Reconciliation`;
|
||||||
`docs/roadmap/catalog.md` so completed behavior is not described only as
|
- `TakeoverMode`;
|
||||||
future work.
|
- `ExistingState`;
|
||||||
|
- `ExistingSharedRoot`;
|
||||||
|
- shared-root owner/output carry fields retained only for old execution.
|
||||||
|
- Remove single-owner and shared-root execution branches from
|
||||||
|
`internal/publish/execute.go`.
|
||||||
|
- Remove unused single-owner/shared-root helper functions from `internal/publish`
|
||||||
|
once no tests or callers use them.
|
||||||
|
- Remove legacy destination config internals from `internal/config`:
|
||||||
|
- `StatePolicy`;
|
||||||
|
- `ReconciliationPolicy`;
|
||||||
|
- `TakeoverPolicy`;
|
||||||
|
- `TransferPolicy`;
|
||||||
|
- legacy constants and defaults that are no longer referenced.
|
||||||
|
- In `internal/state`, keep only what is needed for:
|
||||||
|
- parsing schema version `4` catalog state;
|
||||||
|
- identifying schema versions `1`, `2`, and `3` as superseded legacy by
|
||||||
|
schema number;
|
||||||
|
- validating and writing catalog state;
|
||||||
|
- pruning and reconciling catalog outputs.
|
||||||
|
- Delete or rewrite tests that assert legacy state parsing, validation,
|
||||||
|
comparison, shared-root ownership, or single-owner output behavior.
|
||||||
|
- Preserve test fixtures only where they are used to create superseded legacy
|
||||||
|
state for first catalog-run behavior. Prefer small local helpers over keeping
|
||||||
|
broad legacy state builders.
|
||||||
|
- Remove stale skipped tests that only represent old behavior. If a skipped test
|
||||||
|
still represents current expected behavior, unskip and update it.
|
||||||
|
|
||||||
|
Current-behavior documentation updates:
|
||||||
|
|
||||||
|
- No new user docs should be needed if Stage 2 is complete.
|
||||||
|
- Update internal docs only if removal changes internal package contracts beyond
|
||||||
|
what Stage 2 already documented.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- `go test ./internal/state`
|
||||||
|
- `go test ./internal/publish`
|
||||||
|
- `go test ./internal/app ./internal/cli`
|
||||||
|
- `go test ./internal/config`
|
||||||
|
- `go test ./...`
|
||||||
|
- Consistency searches:
|
||||||
|
- `rg -n "ActionReplaceOlder|ActionReplaceNewer|ActionReplaceConflict|ActionReplaceTakeover|ActionSkipDestinationNewer" internal`
|
||||||
|
- `rg -n "StateModeSingleOwner|StateModeSharedRoot|SharedRootState|DistributorState|ParseSharedRoot|ReconciliationPolicy|TakeoverPolicy|TransferPolicy" internal --glob '!**/*_test.go'`
|
||||||
|
- `rg -n "Retained while the executor is migrated to catalog state" internal`
|
||||||
|
|
||||||
|
Completion criteria:
|
||||||
|
|
||||||
|
- Production publish execution has one catalog code path plus explicit
|
||||||
|
`force_replace`.
|
||||||
|
- Legacy config policy types are gone from production config structs/defaults.
|
||||||
|
- State package no longer exposes full v2/v3 implementation machinery unless it
|
||||||
|
is required by tests that generate superseded legacy fixtures.
|
||||||
|
- Full test suite passes without skipped tests that mask catalog cleanup work.
|
||||||
|
|
||||||
|
## Stage 5: Roadmap Closeout
|
||||||
|
|
||||||
|
Goal: leave `docs/roadmap/` in a clean state after the follow-up work is
|
||||||
|
implemented.
|
||||||
|
|
||||||
|
Implementation scope:
|
||||||
|
|
||||||
|
- Remove or rewrite completed roadmap material:
|
||||||
|
- this `implementation.md`;
|
||||||
|
- `docs/roadmap/catalog.md`, if catalog behavior is fully documented in
|
||||||
|
current docs;
|
||||||
|
- any future roadmap entries that still describe completed catalog cleanup as
|
||||||
|
pending work.
|
||||||
|
- Keep only genuinely future work in `docs/roadmap/future.md` or another active
|
||||||
|
roadmap file.
|
||||||
|
- Ensure future work remains out of current behavior docs.
|
||||||
|
|
||||||
Tests and checks:
|
Tests and checks:
|
||||||
|
|
||||||
- `go test ./...`
|
- `go test ./...`
|
||||||
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
|
- `git status --short`
|
||||||
- `rg -n 'single_owner|shared_root|schema version `2`|schema version `3`' docs internal`
|
- `rg -n "future catalog|planned catalog|single_owner|shared_root|takeover|reconciliation|transfer" docs README.md examples`
|
||||||
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
|
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs README.md examples`
|
||||||
|
|
||||||
Completion criteria:
|
Completion criteria:
|
||||||
|
|
||||||
- Current docs and examples describe the catalog workflow model.
|
- Current docs, examples, and code describe the same implemented behavior.
|
||||||
- Legacy destination policy fields and legacy write paths are gone.
|
- Roadmap docs contain only future work.
|
||||||
- Full test suite passes.
|
- Full test suite passes.
|
||||||
|
|
||||||
## Refactors To Avoid
|
## Refactors To Avoid
|
||||||
|
|
||||||
- Do not add top-level `owners` or `sources` catalogs.
|
- Do not reintroduce `state`, `reconciliation`, `takeover`, or `transfer` YAML
|
||||||
- Do not persist workflow in `.distributor.json`.
|
fields as deprecated aliases.
|
||||||
- Do not keep deprecated config aliases for legacy destination policy fields.
|
- Do not migrate legacy destination state into catalog state.
|
||||||
- Do not implement state migration from schema versions `1`, `2`, or `3`.
|
- Do not add top-level `owners` or `sources` to catalog state.
|
||||||
- Do not add catalog-specific prune/reconcile selectors beyond the agreed
|
- Do not persist `workflow` in `.distributor.json`.
|
||||||
initial ownership scope.
|
- Do not add new prune/reconcile selectors as part of this cleanup.
|
||||||
- Do not move catalog policy into storage adapters.
|
- Do not move catalog or force policy into storage adapters.
|
||||||
|
- Do not redesign the run JSON envelope while cleaning action labels.
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
No open questions are known. The catalog roadmap locks the state shape, workflow
|
No open questions remain. This roadmap locks the follow-up decisions:
|
||||||
values, default workflow, clean-break policy, legacy-state behavior, output
|
catalog `--force` will be implemented for exceptional destructive replacement,
|
||||||
field selection, `created_at` preservation, and maintenance command scope.
|
current docs and examples will be synchronized to catalog schema version `4`,
|
||||||
|
catalog `skip_same` will be implemented as a metadata-only no-op optimization,
|
||||||
|
and legacy single-owner/shared-root implementation debt will be removed after
|
||||||
|
the catalog action vocabulary is complete.
|
||||||
|
|||||||
Reference in New Issue
Block a user