Compare commits
18 Commits
v0.6.0
...
cf7e62733e
| Author | SHA1 | Date | |
|---|---|---|---|
| cf7e62733e | |||
| 91b72478d5 | |||
| 52078e2195 | |||
| 77cde40296 | |||
| 20129bfff5 | |||
| a95662226f | |||
| d23c624179 | |||
| 69043801d0 | |||
| 8b0ce4d134 | |||
| eba65018be | |||
| 11d1eabe2a | |||
| c02106987f | |||
| 598b665307 | |||
| c4e8ebff6f | |||
| 3da7f931b2 | |||
| b10a8bd194 | |||
| c84d8868d1 | |||
| fc33bbca54 |
@@ -292,6 +292,7 @@ Text output is optimized for direct operator use. JSON output is optimized for a
|
||||
- Usage errors and fatal setup errors exit non-zero and do not emit a JSON result document.
|
||||
- `run --format json` emits a JSON result for partial destination failures, sets `ok` to `false`, includes result details and errors, and exits non-zero.
|
||||
- Warnings are included in JSON output and are printed in text output when relevant.
|
||||
- `run` summaries include separate `replace_older`, `replace_conflict`, `replace_newer`, `replace_takeover`, and `force_replace` counters. Takeover destination actions include `takeover_mode` in JSON and text output.
|
||||
|
||||
## Diagnostics And Recovery
|
||||
|
||||
|
||||
@@ -313,6 +313,7 @@ transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
css_href: /assets/report.css
|
||||
```
|
||||
|
||||
`publish.html` controls whether generated HTML outputs are published. When `publish.html` is `true`, `transform.markdown_to_html.enabled` must also be `true`.
|
||||
@@ -322,10 +323,13 @@ Markdown transform fields:
|
||||
- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for this destination.
|
||||
- `transform.markdown_to_html.mode`: optional. Accepted values are `sidecar` and `index`; default is `sidecar` when a Markdown transform block is present.
|
||||
- `transform.markdown_to_html.input`: optional source manifest path for `index` mode only.
|
||||
- `transform.markdown_to_html.css_href`: optional stylesheet href to link from generated HTML.
|
||||
|
||||
`sidecar` mode renders every manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one Markdown source to `index.html` at the destination bundle path. If `index` mode omits `input`, the selected source bundle must contain exactly one Markdown file.
|
||||
|
||||
At least one output type must be enabled. Enabled Markdown transforms are rejected when `publish.html` is `false`, and `input` is rejected unless `mode` is `index`.
|
||||
`css_href` may be an absolute `http` or `https` URL, a root-relative path such as `/assets/report.css`, or a relative URL path such as `assets/report.css`. Query strings are allowed. `distributor` injects the href as a `<link rel="stylesheet">` element but does not copy, publish, verify, or manage the CSS file solely because `css_href` is set.
|
||||
|
||||
At least one output type must be enabled. Enabled Markdown transforms are rejected when `publish.html` is `false`, `input` is rejected unless `mode` is `index`, and `css_href` is rejected when the Markdown transform is disabled.
|
||||
|
||||
## Destination Path Mapping
|
||||
|
||||
@@ -412,7 +416,27 @@ pipelines:
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
Every output path in a shared root belongs to exactly one `pipeline_id` and `destination_id`. A different owner planning the same path fails as a conflict.
|
||||
Every output path in a shared root belongs to exactly one `pipeline_id` and `destination_id`. A different owner planning the same path fails as a conflict unless `takeover.mode` allows the managed output path to move to the current owner.
|
||||
|
||||
## Takeover Policy
|
||||
|
||||
```yaml
|
||||
takeover:
|
||||
mode: same_pipeline
|
||||
```
|
||||
|
||||
- `takeover.mode`: optional. Accepted values are `same_pipeline`, `same_source`, `any_managed`, and `never`; default is `same_pipeline`.
|
||||
|
||||
Takeover controls when a destination may normally replace valid distributor-managed state whose pipeline, destination, source identity, or shared-root output owner differs from the current publication.
|
||||
|
||||
- `same_pipeline`: replace managed state owned by the same pipeline. The previous destination id and source id may differ.
|
||||
- `same_source`: replace managed state only when the existing source manifest id matches the current source id.
|
||||
- `any_managed`: replace any valid distributor-managed state at the selected destination bundle path.
|
||||
- `never`: do not replace identity or source conflicts without the explicit forced replacement workflow.
|
||||
|
||||
For shared-root state, `same_pipeline` permits taking over output paths owned by another destination in the same pipeline, `same_source` permits taking over output paths whose owner records the same source manifest id, and `any_managed` permits taking over output paths owned by any valid shared-root owner. Unrelated owner records and non-conflicting outputs remain managed by their existing owners.
|
||||
|
||||
Takeover does not apply to unmanaged content, invalid destination state, same-created digest conflicts, or same-source destination-newer comparisons.
|
||||
|
||||
## Reconciliation Policy
|
||||
|
||||
@@ -486,7 +510,7 @@ Transfer fields and accepted values:
|
||||
- `transfer.on_destination_newer`: `skip`, `replace`, or `fail`. Default: `skip`.
|
||||
- `transfer.on_conflict`: `fail` or `replace`. Default: `fail`.
|
||||
|
||||
`replace` for `on_destination_newer` and `on_conflict` is honored only when `run --force` is supplied. There is no config field that enables forced replacement by default.
|
||||
`replace` for `on_destination_newer` and `on_conflict` applies only to valid distributor-managed destination state. Unmanaged destination content and invalid destination state still require the explicit `run --force` workflow where supported.
|
||||
|
||||
## Size And Duration Values
|
||||
|
||||
@@ -525,6 +549,7 @@ Defaults are applied after YAML decoding and before validation:
|
||||
- `links.primary: auto` when a `links` block is present and `primary` is omitted
|
||||
- `state.mode: single_owner`
|
||||
- `reconciliation.mode: replace`
|
||||
- `takeover.mode: same_pipeline`
|
||||
- `retention.prune.enabled: false`
|
||||
- `transfer.on_destination_same: skip`
|
||||
- `transfer.on_destination_older: replace`
|
||||
|
||||
@@ -96,9 +96,11 @@ Destination reconciliation applies when destination state is older than the sour
|
||||
- `replace`: for single-owner state, delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set. For shared-root state, delete only the current owner's omitted outputs and preserve unrelated owners.
|
||||
- `merge`: retain managed output paths omitted from the new plan, overwrite planned paths only when they are already recorded in existing state, fail when a newly planned path exists in storage but is not recorded as managed, and write state whose `outputs` are the cumulative managed set.
|
||||
|
||||
Takeover replacement applies when valid managed state has an identity, source, or shared-root output-owner mismatch and destination `takeover.mode` permits the current publication to take ownership. It uses bounded managed replacement behavior. For shared-root state, only taken-over output records and the current owner records are rewritten; unrelated owner records and non-conflicting outputs remain managed by their existing owners.
|
||||
|
||||
Top-level `links.primary_url` is selected from the newly planned outputs for the current publication. Retained outputs keep their existing per-output URL metadata.
|
||||
|
||||
If a merge publication fails after writing outputs, cleanup removes only newly created outputs from that failed attempt. Previously managed overwritten paths remain managed and are not removed by failed-attempt cleanup.
|
||||
If a same-source merge publication fails after writing outputs, cleanup removes only newly created outputs from that failed attempt. Previously managed overwritten paths remain managed and are not removed by failed-attempt cleanup. Takeover replacement does not retain omitted outputs through merge reconciliation.
|
||||
|
||||
## Comparison Semantics
|
||||
|
||||
@@ -106,15 +108,16 @@ If a merge publication fails after writing outputs, cleanup removes only newly c
|
||||
|
||||
- No state and no content: publish new outputs.
|
||||
- No state and existing content: treat the destination as unmanaged.
|
||||
- Shared-root state without the current owner: publish new outputs for that owner if planned paths do not collide with other owners or unmanaged content.
|
||||
- Shared-root state without the current owner: publish new outputs for that owner if planned paths do not collide with unmanaged content or with other owners that `takeover.mode` does not permit.
|
||||
- Matching embedded source manifest: skip.
|
||||
- Same source id with older `created`: replace if policy allows.
|
||||
- Same source id with newer `created`: skip by default.
|
||||
- Same source id and same `created` with different digest: conflict.
|
||||
- Different source id, pipeline id, or destination id: conflict.
|
||||
- Same source id with newer `created`: skip by default, or replace when `transfer.on_destination_newer: replace` is configured.
|
||||
- Same source id and same `created` with different digest: conflict by default, or replace when `transfer.on_conflict: replace` is configured.
|
||||
- Different source id, pipeline id, or destination id in single-owner state: conflict unless `takeover.mode` permits managed ownership transfer or `transfer.on_conflict: replace` is configured.
|
||||
- Shared-root output path owned by a different owner: conflict unless `takeover.mode` permits managed ownership transfer or `transfer.on_conflict: replace` is configured.
|
||||
- Invalid state JSON or invalid state fields: conflict.
|
||||
|
||||
Normal single-owner replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Shared-root replacement deletes only omitted outputs for the current owner. Merge publication retains omitted managed outputs. Forced replacement deletes the bounded destination bundle path.
|
||||
Normal single-owner replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Shared-root replacement deletes only omitted outputs for the current owner. Merge publication retains omitted managed outputs for same-source replacement. Forced replacement deletes the bounded destination bundle path.
|
||||
|
||||
## State Repair Semantics
|
||||
|
||||
@@ -238,7 +241,7 @@ For shared-root state:
|
||||
|
||||
If `state.mode: shared_root` is configured and existing state is a compatible single-owner `.distributor.json` for the same pipeline id and destination id, the next successful publish writes schema version `3` shared-root state for that owner.
|
||||
|
||||
If existing single-owner state belongs to a different pipeline or destination, publish fails as a conflict. `distributor` does not implicitly take over unrelated state or unmanaged files.
|
||||
If `state.mode: shared_root` is configured and existing single-owner state belongs to a different pipeline or destination, publish fails as a conflict. `distributor` does not implicitly convert unrelated single-owner state or take over unmanaged files during shared-root migration.
|
||||
|
||||
## Boundaries
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Rendering uses `github.com/yuin/goldmark`. The exact version is pinned in `go.mo
|
||||
|
||||
## Renderer Behavior
|
||||
|
||||
The transformer constructs `goldmark.New()` with no project-specific extensions, parser options, renderer options, templates, CSS, or metadata injection.
|
||||
The transformer constructs `goldmark.New()` with no project-specific extensions, parser options, renderer options, templates, or source manifest metadata injection.
|
||||
|
||||
Supported output modes:
|
||||
|
||||
@@ -19,6 +19,8 @@ Supported output modes:
|
||||
|
||||
In `index` mode, `transform.markdown_to_html.input` may name the source manifest path to render. If `input` is omitted, the source manifest must list exactly one `.md` file. The selected input must be a clean relative source path, must be listed in the source manifest, and must end in `.md`.
|
||||
|
||||
When `transform.markdown_to_html.css_href` is set, generated HTML includes a stylesheet link in the document head. The href may be an absolute HTTP(S) URL, a root-relative path, or a relative URL path. Distributor treats this as a link reference only; it does not copy, publish, verify, or manage the CSS file solely because `css_href` is configured.
|
||||
|
||||
Raw HTML embedded in Markdown is not passed through by the current renderer behavior. Tests allow Goldmark's disabled-or-escaped raw HTML output forms and reject literal script tags in generated HTML.
|
||||
|
||||
## HTML Wrapper
|
||||
@@ -28,10 +30,11 @@ Rendered Markdown body HTML is wrapped in a fixed document shell:
|
||||
- `<!doctype html>`
|
||||
- `<html lang="en">`
|
||||
- UTF-8 `<meta charset>`
|
||||
- optional `<link rel="stylesheet" href="...">` when `css_href` is configured
|
||||
- empty `<title>`
|
||||
- `<body>` containing the rendered Markdown body
|
||||
|
||||
The wrapper is deterministic and does not read configuration, templates, CSS, or source manifest metadata.
|
||||
The wrapper is deterministic. When `css_href` is omitted, the generated wrapper is unchanged from the unstyled output. When `css_href` is configured, its escaped link element is part of the generated output bytes.
|
||||
|
||||
## Output Metadata
|
||||
|
||||
|
||||
@@ -52,7 +52,7 @@ The adapter uses these S3 operations:
|
||||
|
||||
Writes buffer the input and set `ContentLength`. If no content type is supplied by the caller, the adapter infers a content type from the logical path.
|
||||
|
||||
Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.
|
||||
Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Same-source merge publication retains omitted managed objects and may overwrite existing managed objects. Takeover replacement does not retain omitted outputs through merge reconciliation. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.
|
||||
|
||||
## Error Mapping
|
||||
|
||||
|
||||
@@ -54,7 +54,7 @@ The configured `path` is the backend root. All source discovery, destination pat
|
||||
|
||||
The adapter rejects symlink ancestors for reads and writes. Reads require regular files. Writes create parent directories and prefer atomic temp-file-plus-rename writes when requested. Walk output is sorted through the shared storage walker.
|
||||
|
||||
Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Merge publication retains omitted managed paths and may overwrite existing managed paths. Forced replacement deletes the bounded destination bundle path.
|
||||
Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Same-source merge publication retains omitted managed paths and may overwrite existing managed paths. Takeover replacement does not retain omitted outputs through merge reconciliation. Forced replacement deletes the bounded destination bundle path.
|
||||
|
||||
## Boundaries
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ User-facing command parsing stays in `internal/cli`, including `reconcile-state`
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, state policy, reconciliation policy, retention policy, and transfer policy.
|
||||
The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, state policy, reconciliation policy, takeover policy, retention policy, and transfer policy.
|
||||
|
||||
Config fields are validated and defaulted by `internal/config` before app workflows use them.
|
||||
|
||||
@@ -32,7 +32,7 @@ The app layer registers default transforms, including Markdown-to-HTML, and supp
|
||||
|
||||
## State And Manifest Behavior
|
||||
|
||||
Run workflows discover and validate source bundles through `internal/bundle`. Destination state actions are prepared and written through `internal/publish` and `internal/state`; the app layer records report projections of those actions and results.
|
||||
Run workflows discover and validate source bundles through `internal/bundle`. Destination state actions are prepared and written through `internal/publish` and `internal/state`; the app layer records report projections of those actions and results. Run summaries count older-state, conflict, newer-state, takeover, and explicit forced replacements separately.
|
||||
|
||||
Reconcile-state workflows load one configured pipeline/destination selector, open that destination root, parse the root `.distributor.json`, and report missing managed output records plus unmanaged storage entries. Managed output existence checks use storage `Stat`; unmanaged reporting uses bounded storage `Walk` and excludes `.distributor.json` plus all paths already recorded as managed. Apply mode removes missing managed output records from state and rewrites valid state only; dry-run reports the same repair without writing. Text output reports `changed`, `would_change`, or `unchanged`; JSON output uses the shared app envelope. It does not validate output digests, delete destination files, adopt unmanaged files, or rewrite invalid or mismatched state.
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ The canonical user-facing config reference is `docs/config.md`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
The package defines all user-visible config fields: `server.http`, `secrets`, `pipelines`, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, state policy, reconciliation policy, retention policy, and transfer policy.
|
||||
The package defines all user-visible config fields: `server.http`, `secrets`, `pipelines`, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, state policy, reconciliation policy, takeover policy, retention policy, and transfer policy.
|
||||
|
||||
## Adapters Used
|
||||
|
||||
@@ -26,7 +26,7 @@ No external storage adapters are used directly. The package exposes normalized c
|
||||
|
||||
## State And Manifest Behavior
|
||||
|
||||
The package does not parse source manifests or destination state. It validates config values that later affect manifest validation and destination state, such as publish/transform combinations, links, state policy, reconciliation policy, retention policy, transfer policy, backend roots, S3 prefix shape, and HTTP upload source settings.
|
||||
The package does not parse source manifests or destination state. It validates config values that later affect manifest validation and destination state, such as publish/transform combinations, links, state policy, reconciliation policy, takeover policy, retention policy, transfer policy, backend roots, S3 prefix shape, and HTTP upload source settings.
|
||||
|
||||
## Skip And Resume Behavior
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ Audience: developers and LLM coding agents changing `internal/publish`.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, state policy, reconciliation policy, transformer resolver, transfer policy, distributor version, and force flag.
|
||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, state policy, reconciliation policy, takeover policy, transformer resolver, transfer policy, distributor version, and force flag.
|
||||
|
||||
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, current-owner outputs retained by merge, current-owner outputs deleted by replace, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
|
||||
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, takeover mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, taken-over outputs, current-owner outputs retained by merge, current-owner outputs deleted by replace or takeover, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
|
||||
|
||||
## Boundaries
|
||||
|
||||
@@ -20,7 +20,7 @@ External destination state semantics are documented in `docs/integrations/destin
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
The package consumes already-defaulted config values for destination `publish`, `transform`, `links`, `state`, `reconciliation`, `transfer`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
|
||||
The package consumes already-defaulted config values for destination `publish`, `transform`, `links`, `state`, `reconciliation`, `takeover`, `transfer`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
|
||||
|
||||
## Adapters Used
|
||||
|
||||
@@ -28,25 +28,25 @@ The package depends on `internal/storage.Backend` for source and destination IO,
|
||||
|
||||
## State And Manifest Behavior
|
||||
|
||||
Planning inspects destination state through `internal/state`, compares it with the source manifest, and maps comparison outcomes plus transfer policy into actions: `publish_new`, `replace_older`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, or `fail_unmanaged`.
|
||||
Planning inspects destination state through `internal/state`, compares it with the source manifest, and maps comparison outcomes plus transfer and takeover policy into actions: `publish_new`, `replace_older`, `replace_conflict`, `replace_newer`, `replace_takeover`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, or `fail_unmanaged`.
|
||||
|
||||
Single-owner destinations compare the whole destination state against the configured pipeline and destination ids. Shared-root destinations compare only the current owner scope, keyed by pipeline id and destination id. An absent shared-root owner is publishable for that owner unless a planned output collides with unmanaged storage content. Planned writes to a path owned by another shared-root owner fail as conflicts.
|
||||
Single-owner destinations compare the whole destination state against the configured pipeline and destination ids. Valid managed identity and source conflicts can become `replace_takeover` when `takeover.mode` allows them. Shared-root destinations compare only the current owner scope, keyed by pipeline id and destination id. An absent shared-root owner is publishable for that owner unless a planned output collides with unmanaged storage content. Planned writes to a path owned by another shared-root owner become `replace_takeover` when `takeover.mode` allows that managed output path to move to the current owner.
|
||||
|
||||
Execution writes destination state after selected outputs are written. Destination state includes copied source output metadata, generated output metadata, output timestamps, embedded source manifest, reconciliation metadata, link metadata when configured, pipeline id, destination id, and publication timestamps.
|
||||
|
||||
## Skip And Resume Behavior
|
||||
|
||||
`skip_same` and `skip_destination_newer` execute as no-ops. Replacement-mode single-owner updates remove managed output paths from existing state plus `.distributor.json`, verify the destination is empty, and write state whose outputs are exactly the new plan. Replacement-mode shared-root updates remove only current-owner omitted outputs and preserve unrelated owners. Merge-mode updates retain omitted managed outputs, overwrite only paths already recorded as managed, reject unmanaged destination path collisions, and write cumulative output state. Failed writes trigger cleanup where practical; merge cleanup removes only newly created outputs from the failed attempt.
|
||||
`skip_same` and `skip_destination_newer` execute as no-ops. Replacement-mode single-owner updates remove managed output paths from existing state plus `.distributor.json`, verify the destination is empty, and write state whose outputs are exactly the new plan. `replace_conflict` uses managed replacement mechanics and does not retain omitted outputs through merge reconciliation. `replace_newer` follows the same managed replacement and merge-retention rules as `replace_older`. `replace_takeover` uses managed replacement mechanics and does not retain omitted outputs through merge reconciliation. Replacement-mode shared-root updates remove only current-owner omitted outputs and preserve unrelated owners. Shared-root takeover rewrites only the taken-over output records and current owner records. Merge-mode same-source updates retain omitted managed outputs, overwrite only paths already recorded as managed, reject unmanaged destination path collisions, and write cumulative output state. Failed writes trigger cleanup where practical; same-source merge cleanup removes only newly created outputs from the failed attempt.
|
||||
|
||||
Shared-root execution writes schema version `3` state. It preserves unrelated owner records and outputs, updates only the publishing owner metadata, preserves root `created_at`, and updates root `updated_at` after successful state writes. Compatible single-owner state for the same pipeline and destination is converted to shared-root state on successful publish.
|
||||
|
||||
Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state.
|
||||
Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state. Valid managed conflict and newer-state replacements are normal managed replacement actions when transfer policy allows them; they are not forced replacement actions.
|
||||
|
||||
Retention pruning is not part of publish execution and does not run automatically after a successful publish. The app-level prune workflow uses destination state after publication to select managed outputs for deletion.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Planning fails for incomplete requests, invalid publish/transform policy, invalid state mode, invalid reconciliation mode, output path collisions, invalid destination state, unmanaged destination content without force, shared-root owner path conflicts, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
|
||||
Planning fails for incomplete requests, invalid publish/transform policy, invalid state mode, invalid reconciliation mode, output path collisions, invalid destination state, unmanaged destination content without force, shared-root owner path conflicts not allowed by `takeover.mode`, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
|
||||
|
||||
Execution fails on delete, read, transform output, unmanaged merge path collision, shared-root ownership conflict, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
|
||||
|
||||
@@ -65,7 +65,7 @@ Execution fails on delete, read, transform output, unmanaged merge path collisio
|
||||
- Replacement reconciliation deletes only managed paths recorded in existing state plus `.distributor.json` for single-owner state, and only current-owner omitted outputs for shared-root state.
|
||||
- Merge reconciliation never adopts unmanaged content.
|
||||
- Merge state output records are cumulative for the single owner.
|
||||
- Shared-root planning is owner-scoped and preserves unrelated owner outputs.
|
||||
- Shared-root planning is owner-scoped, preserves unrelated owner outputs, and records taken-over managed output paths separately from unrelated owners.
|
||||
- Shared-root execution writes owner-scoped changes without deleting unrelated owners.
|
||||
- Forced replacement deletes only within the supplied destination bundle path.
|
||||
- Destination state is written after selected outputs are written.
|
||||
|
||||
@@ -44,7 +44,7 @@ Shared-root helper projections preserve output `created_at` for existing managed
|
||||
|
||||
## Skip And Resume Behavior
|
||||
|
||||
Comparison is pure. It returns outcomes for absent state, unmanaged content, invalid state, pipeline/destination mismatch, same source manifest, older destination, newer destination, same-created digest conflict, and different source id conflict. It does not decide whether to skip, replace, force, or fail; publish planning maps outcomes to actions.
|
||||
Comparison is pure. It returns outcomes for absent state, unmanaged content, invalid state, pipeline/destination mismatch, same source manifest, older destination, newer destination, same-created digest conflict, and different source id conflict. Shared-root helpers expose structured output ownership conflicts. State code does not decide whether to skip, replace, take over, force, or fail; publish planning maps outcomes and conflicts to actions.
|
||||
|
||||
Shared-root comparison is owner-scoped. It compares only the owner keyed by the current pipeline id and destination id, treats an absent owner as absent destination state for that owner, and can compare compatible single-owner state for the current owner without converting unrelated single-owner state.
|
||||
|
||||
@@ -74,7 +74,7 @@ Parsing rejects invalid JSON, trailing data, missing required fields, invalid ti
|
||||
- Newly written single-owner state uses schema version `2`.
|
||||
- Schema version `1` state remains readable as replacement-mode single-owner state.
|
||||
- Schema version `3` shared-root state is parsed and validated without converting unrelated single-owner state.
|
||||
- Shared-root owner updates preserve unrelated owners and reject planned path collisions with other owners.
|
||||
- Shared-root owner updates preserve unrelated owners. Publish planning removes taken-over output records before owner replacement when takeover policy allows a managed path to move owners.
|
||||
- Missing-output repair helpers preserve unrelated owner records and outputs.
|
||||
- Prune planning uses output `updated_at` and preserves unrelated shared-root owners.
|
||||
- Generated outputs always record a transform id.
|
||||
|
||||
@@ -67,11 +67,16 @@ Published destination bundle paths contain `.distributor.json`. See [Destination
|
||||
- No destination state and no destination content: publish new outputs.
|
||||
- Matching destination state: skip as already published.
|
||||
- Older destination state for the same source id: replace if transfer policy allows it.
|
||||
- Newer destination state: skip by default.
|
||||
- Invalid destination state, identity mismatch, different source id, or same-created digest mismatch: fail by default.
|
||||
- Newer destination state for the same source id: skip by default, or replace when `transfer.on_destination_newer: replace` is configured.
|
||||
- Valid managed state with an identity, source, same-created digest, or shared-root output-owner mismatch: replace when destination `takeover.mode` allows it or, for remaining valid managed conflicts, when `transfer.on_conflict: replace` is configured.
|
||||
- Invalid destination state, identity, source, same-created digest, or shared-root output-owner mismatches not allowed by `takeover.mode` or `transfer.on_conflict: replace`: fail by default.
|
||||
- Content without `.distributor.json`: fail as unmanaged content by default.
|
||||
|
||||
When destination state is older than the source, `transfer.on_destination_older` controls whether publication may proceed and `reconciliation.mode` controls how managed outputs are updated.
|
||||
When destination state is older or newer than the source, `transfer.on_destination_older` and `transfer.on_destination_newer` control whether publication may proceed and `reconciliation.mode` controls how managed outputs are updated.
|
||||
|
||||
When `transfer.on_conflict: replace` permits `replace_conflict`, the destination is valid managed state but does not match the current source, pipeline, destination, or same-created digest. `replace_conflict` rewrites the affected managed destination state and does not retain omitted outputs through merge reconciliation.
|
||||
|
||||
For takeover replacement, `reconciliation.mode: merge` does not retain omitted outputs from the previous source identity. The destination is rewritten as a managed replacement for the current source or shared-root owner.
|
||||
|
||||
`reconciliation.mode: replace` is the default. It deletes only managed output paths recorded in `.distributor.json` plus the state file, verifies the destination bundle path is empty, then writes the newly planned outputs and state. The new state `outputs` array is exactly the newly planned output set.
|
||||
|
||||
@@ -79,11 +84,11 @@ When destination state is older than the source, `transfer.on_destination_older`
|
||||
|
||||
For both modes, retained or overwritten paths are identified only from `.distributor.json`; unmanaged files are not adopted.
|
||||
|
||||
For `state.mode: shared_root`, one destination root may contain outputs from multiple pipeline/destination owners. Comparisons, replacement, and merge retention are scoped to the current owner. Outputs owned by other owners are preserved. A planned output path owned by another owner fails as a conflict, and a planned path that exists in storage but is not recorded in state fails as unmanaged content by default.
|
||||
For `state.mode: shared_root`, one destination root may contain outputs from multiple pipeline/destination owners. Comparisons, replacement, and merge retention are scoped to the current owner. Outputs owned by other owners are preserved unless a planned output path is owned by another valid owner and `takeover.mode` allows moving that path to the current owner. A planned path that exists in storage but is not recorded in state fails as unmanaged content by default.
|
||||
|
||||
If `state.mode: shared_root` is configured on a destination whose existing single-owner state belongs to the same pipeline and destination, the next successful publish converts that state file to shared-root schema. Existing single-owner state for a different pipeline or destination remains a conflict.
|
||||
|
||||
If a write fails after some outputs were written, `distributor` attempts cleanup before returning the error. In `replace` mode, cleanup removes outputs written during that failed attempt. In `merge` mode, cleanup removes only newly created outputs from that failed attempt; overwritten managed outputs are left in place because they previously belonged to the managed set. Operators should still inspect the destination after a failed write before retrying.
|
||||
If a write fails after some outputs were written, `distributor` attempts cleanup before returning the error. In `replace` mode and takeover replacement, cleanup removes outputs written during that failed attempt. In same-source `merge` mode, cleanup removes only newly created outputs from that failed attempt; overwritten managed outputs are left in place because they previously belonged to the managed set. Operators should still inspect the destination after a failed write before retrying.
|
||||
|
||||
Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.
|
||||
|
||||
@@ -150,12 +155,15 @@ For single-owner state, the state owner must match the selected pipeline and des
|
||||
|
||||
`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries. For reconciliation, dry runs report the same high-level action labels as execution; inspect the configured destination's `reconciliation.mode` to determine whether `replace_older` will replace the managed set or merge into it.
|
||||
|
||||
For shared-root destinations, dry runs are owner-scoped. A `replace_older` action replaces or merges only the current owner according to `reconciliation.mode`; unrelated owners remain managed by the shared-root state.
|
||||
For shared-root destinations, dry runs are owner-scoped. A `replace_older` action replaces or merges only the current owner according to `reconciliation.mode`; unrelated owners remain managed by the shared-root state. Paths owned by another owner fail as conflicts unless `takeover.mode` allows ownership transfer.
|
||||
|
||||
Review these action labels before publishing:
|
||||
|
||||
- `publish_new`: destination state is absent, or a shared-root owner is absent and planned paths are publishable.
|
||||
- `replace_older`: destination state is older than the source.
|
||||
- `replace_newer`: destination state is newer than the source and `transfer.on_destination_newer: replace` allows managed replacement.
|
||||
- `replace_conflict`: destination state is valid managed state with a conflict and `transfer.on_conflict: replace` allows managed replacement.
|
||||
- `replace_takeover`: destination state is valid managed state and `takeover.mode` allows replacement across an identity, source, or shared-root output-owner mismatch.
|
||||
- `skip_same`: destination state already matches the source.
|
||||
- `skip_destination_newer`: destination state is newer than the source and is skipped.
|
||||
- `force_replace`: destructive replacement selected because `--force` is present and policy permits it.
|
||||
@@ -163,7 +171,7 @@ Review these action labels before publishing:
|
||||
|
||||
Fixed destinations add fixed-path warnings during dry runs, including the selected source bundle and replacement warnings when the destination root would be replaced.
|
||||
|
||||
JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.
|
||||
Text and JSON summaries count `publish_new`, `replace_older`, `replace_conflict`, `replace_newer`, `replace_takeover`, `force_replace`, skipped, and failed destinations separately. Takeover action records include the configured takeover mode and the conflict reason. JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.
|
||||
|
||||
## Forced Replacement Workflow
|
||||
|
||||
@@ -174,10 +182,7 @@ go run ./cmd/distributor run --config <config-path> --dry-run --force
|
||||
go run ./cmd/distributor run --config <config-path> --force
|
||||
```
|
||||
|
||||
Forced replacement can claim unmanaged non-empty destination paths. State conflicts require both `--force` and transfer policy that permits replacement:
|
||||
|
||||
- newer destination state requires `transfer.on_destination_newer: replace`;
|
||||
- conflict outcomes require `transfer.on_conflict: replace`.
|
||||
Forced replacement can claim unmanaged non-empty destination paths and is reserved for exceptional destructive replacement. Valid managed newer state and valid managed conflict state use `replace_newer` and `replace_conflict` when the corresponding transfer policy allows replacement; they do not require `--force`.
|
||||
|
||||
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root.
|
||||
|
||||
@@ -267,7 +272,7 @@ S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration]
|
||||
|
||||
When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
|
||||
|
||||
Normal single-owner replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Shared-root replacement deletes only current-owner omitted output objects and rewrites the shared state object. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
|
||||
Normal single-owner replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Shared-root replacement deletes only current-owner omitted output objects and rewrites the shared state object. Same-source merge publication retains omitted managed objects and may overwrite existing managed objects. Takeover replacement does not retain omitted outputs through merge reconciliation. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
|
||||
|
||||
## Secrets Operation
|
||||
|
||||
@@ -282,7 +287,7 @@ Use these recovery boundaries:
|
||||
- For source validation failures, regenerate the source bundle and manifest together.
|
||||
- For an empty or missing destination, rerun after fixing config or storage access.
|
||||
- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
|
||||
- For shared-root ownership conflicts, change one owner so it writes a different destination path, or use a separate destination root.
|
||||
- For shared-root ownership conflicts, change one owner so it writes a different destination path, use a separate destination root, or configure `takeover.mode` when the current owner should take over valid managed output paths.
|
||||
- For missing managed output files recorded in state, run `reconcile-state --dry-run`, then apply `reconcile-state` if the missing files should no longer be considered managed.
|
||||
- For configured retention cleanup, run `prune --dry-run`, then apply `prune --apply` after reviewing the managed output list.
|
||||
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`. In merge mode, retained outputs may be intentional managed outputs from the prior state.
|
||||
|
||||
@@ -169,9 +169,9 @@ Destination comparison rules are based on `.distributor.json`:
|
||||
- No `.distributor.json`: publish normally only if the destination bundle path is empty.
|
||||
- Existing state embeds the same normalized source manifest: skip as already published.
|
||||
- Existing state has the same source id and an older source `created`: replace, subject to destructive-operation safety rules.
|
||||
- Existing state has the same source id and a newer source `created`: skip because the destination is newer than the source.
|
||||
- Existing state has the same source id and same `created` but different digest: fail as a conflict.
|
||||
- Existing state has a different source id: fail as a conflict.
|
||||
- Existing state has the same source id and a newer source `created`: skip by default, or replace when `transfer.on_destination_newer: replace` is configured.
|
||||
- Existing state has the same source id and same `created` but different digest: fail by default, or replace as valid managed conflict state when `transfer.on_conflict: replace` is configured.
|
||||
- Existing valid managed state has a different source id, pipeline id, destination id, or shared-root output owner: replace when destination `takeover.mode` permits that ownership transfer, or replace as valid managed conflict state when `transfer.on_conflict: replace` is configured; otherwise fail as a conflict.
|
||||
|
||||
For older destination state, destination `reconciliation.mode` controls output updates. `replace` rewrites the managed output set to match the new plan. `merge` retains omitted managed outputs, overwrites only existing managed paths, and rejects planned paths that collide with unmanaged storage content.
|
||||
|
||||
@@ -264,6 +264,7 @@ Pipeline configuration should express:
|
||||
- per-destination transform policy;
|
||||
- per-destination public link policy;
|
||||
- validation behavior;
|
||||
- per-destination takeover behavior;
|
||||
- destination conflict/replacement behavior.
|
||||
|
||||
## Modules and Registries
|
||||
|
||||
@@ -18,7 +18,7 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
|
||||
- `internal/adapters/ssh`: SSH/SFTP backend.
|
||||
- `internal/adapters/s3`: S3-compatible object storage backend.
|
||||
- `internal/storage/fake`: in-memory backend for tests.
|
||||
- `internal/publish`: destination inspection, output planning, reconciliation, execution, managed cleanup, and explicit forced replacement.
|
||||
- `internal/publish`: destination inspection, output planning, takeover planning, reconciliation, execution, managed cleanup, and explicit forced replacement.
|
||||
- `internal/transform`: transform interface and registry.
|
||||
- `internal/transform/markdown`: Markdown-to-HTML transform.
|
||||
- `internal/notify`: notification interface and current no-op notifier.
|
||||
|
||||
@@ -43,6 +43,7 @@ Canonical homes:
|
||||
|
||||
- project purpose and quickstart: `README.md`
|
||||
- development principles: `docs/policy/architecture.md`
|
||||
- public HTTP API reference: `docs/api.md`
|
||||
- configuration reference: `docs/config.md`
|
||||
- CLI reference: `docs/cli.md`
|
||||
- operations and recovery: `docs/operations.md`
|
||||
@@ -122,6 +123,22 @@ Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- validated examples under `examples/`
|
||||
|
||||
### Public HTTP API service
|
||||
|
||||
Required:
|
||||
- `docs/api.md`
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/consumers/`, for task-oriented client integration guides
|
||||
- `docs/integrations/`, for upstream/downstream service contracts
|
||||
- validated examples under `examples/`
|
||||
|
||||
### Project with public packages or consumer APIs
|
||||
|
||||
Required:
|
||||
@@ -173,6 +190,32 @@ It should include:
|
||||
|
||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||
|
||||
### docs/api.md
|
||||
|
||||
**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
|
||||
|
||||
Required for projects whose primary public interface is HTTP.
|
||||
|
||||
`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
|
||||
|
||||
It should include:
|
||||
|
||||
1. base URL conventions;
|
||||
2. authentication and authorization behavior, if implemented;
|
||||
3. response envelope;
|
||||
4. supported media types and content negotiation behavior;
|
||||
5. shared query parameters;
|
||||
6. endpoint reference grouped by route family;
|
||||
7. request parameters and validation rules;
|
||||
8. response fields, units, nullability, and optionality;
|
||||
9. error response shape and status codes;
|
||||
10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
|
||||
11. compact request and response examples.
|
||||
|
||||
It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
|
||||
|
||||
For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
|
||||
|
||||
### docs/policy/development.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
@@ -264,6 +307,8 @@ Required for projects with public packages, SDKs, client APIs, plugin APIs, or o
|
||||
|
||||
This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
|
||||
|
||||
For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
|
||||
|
||||
`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
|
||||
|
||||
1. intended consumer audience and use cases;
|
||||
@@ -330,6 +375,8 @@ Required for projects that depend on external CLIs, APIs, services, protocols, o
|
||||
|
||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
|
||||
|
||||
For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
|
||||
|
||||
Use one file per integration where useful.
|
||||
|
||||
## Examples Directory
|
||||
@@ -385,9 +432,10 @@ Before merging documentation changes, verify:
|
||||
|
||||
- README is concise and orientation-focused.
|
||||
- `docs/policy/architecture.md` describes development principles.
|
||||
- `docs/api.md` is the canonical HTTP contract for HTTP API services.
|
||||
- Future work appears only under `docs/roadmap/`.
|
||||
- User-facing docs avoid unnecessary internals.
|
||||
- Consumer-facing docs explain public APIs without duplicating integration contracts.
|
||||
- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
|
||||
- Developer-facing docs preserve boundaries and invariants.
|
||||
- Config examples match the schema.
|
||||
- CLI examples match real commands and flags.
|
||||
|
||||
283
docs/roadmap/catalog.md
Normal file
283
docs/roadmap/catalog.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# Catalog State And Destination Workflow Roadmap
|
||||
|
||||
This roadmap records the intended first-class state model and destination
|
||||
workflow configuration for replacement and additive publication workflows.
|
||||
|
||||
Current `single_owner` and `shared_root` state are oriented around comparing a
|
||||
destination owner to one latest source manifest. That works well for replacement
|
||||
workflows, where a producer maintains a curated source of truth and
|
||||
`distributor` makes a destination match it. It is a poor fit for additive
|
||||
workflows, where each run contributes new or updated output paths while
|
||||
preserving unrelated managed outputs from previous runs and other pipelines.
|
||||
|
||||
The target model is a current-state catalog: `.distributor.json` records the
|
||||
currently managed output paths at a destination root, and each output record
|
||||
stores its current owner, compact source identity, content digest, and update
|
||||
metadata. It is not intended to be an audit log.
|
||||
|
||||
Both replacement and additive workflows should use the same catalog state shape.
|
||||
The workflow choice is runtime policy from destination config, not persisted
|
||||
state.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
- Add one destination state mode named `catalog` for both replacement and
|
||||
additive workflows.
|
||||
- Catalog state writes `.distributor.json` schema version `4`.
|
||||
- Add a destination-level `workflow` setting with accepted values
|
||||
`replacement` and `additive`.
|
||||
- Default `workflow` to `additive` because it is the least destructive workflow.
|
||||
- `workflow` replaces the normal user-facing need to combine `state.mode`,
|
||||
`reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the
|
||||
two primary workflows.
|
||||
- `workflow: replacement` treats the current planned outputs as the authoritative
|
||||
managed output set for the destination scope and removes previously managed
|
||||
outputs in that scope when they are no longer planned.
|
||||
- `workflow: additive` writes or overwrites the planned output paths and retains
|
||||
unrelated managed outputs.
|
||||
- If a planned path already exists as a managed output, the new publication may
|
||||
overwrite it and becomes that path's current owner.
|
||||
- If a planned path exists in storage but is not recorded in valid catalog state,
|
||||
it remains unmanaged content and must not be adopted implicitly.
|
||||
- If an output path changes owner, preserve the output record's existing
|
||||
`created_at` and update only `updated_at`.
|
||||
- Catalog state records current ownership only. It does not retain historical
|
||||
owners, historical sources, or old versions of overwritten output records.
|
||||
- `pipeline_id` and `destination_id` are stored separately on each output
|
||||
record. They are not concatenated into one owner string.
|
||||
- Catalog state does not include top-level `owners`; owners are derivable from
|
||||
the output records.
|
||||
- Catalog state does not include top-level `sources`; compact source identity is
|
||||
stored directly on each output record.
|
||||
- Catalog state does not record whether the last run used `replacement` or
|
||||
`additive`. Workflow is execution policy, and persisting it would create
|
||||
drift risk if config changes later.
|
||||
- Legacy destination policy fields should be rejected outright in the new
|
||||
workflow config model. Do not retain aliases for `state`, `reconciliation`,
|
||||
`takeover`, or `transfer`.
|
||||
|
||||
## Destination Workflow Semantics
|
||||
|
||||
The user-facing destination config should express intent directly:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-latest
|
||||
backend: local
|
||||
path: /srv/reports/weather/latest
|
||||
workflow: additive
|
||||
```
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-archive
|
||||
backend: local
|
||||
path: /srv/reports/weather/archive
|
||||
workflow: replacement
|
||||
```
|
||||
|
||||
`workflow` is orthogonal to backend config, path mapping, publish source/HTML
|
||||
selection, transforms, links, and retention.
|
||||
|
||||
### Replacement Workflow
|
||||
|
||||
A replacement destination is for producers that maintain a curated source of
|
||||
truth and expect `distributor` to make the destination's managed scope match the
|
||||
current publication.
|
||||
|
||||
For a destination configured with `workflow: replacement`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each written output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs in the destination scope that are not present in the current
|
||||
plan are deleted and removed from catalog state;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- the catalog state shape remains the same as additive workflow state.
|
||||
|
||||
### Additive Workflow
|
||||
|
||||
An additive destination is for producers that routinely contribute outputs to a
|
||||
shared destination root.
|
||||
|
||||
For a destination configured with `workflow: additive`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each overwritten output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs not present in the current plan are retained;
|
||||
- unrelated managed outputs from other pipelines or destinations are retained;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- pruning can select retained managed outputs by `updated_at` and, when useful,
|
||||
by `pipeline_id` and/or `destination_id`.
|
||||
|
||||
Replacement workflow treats the current publication as the desired managed
|
||||
output set for a destination scope. Additive workflow treats the current
|
||||
publication as a patch to the catalog of currently managed outputs.
|
||||
|
||||
## Destination State Schema Version 4
|
||||
|
||||
Catalog state should use this top-level shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 4,
|
||||
"distributor_version": "dev",
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"state": {
|
||||
"mode": "catalog"
|
||||
},
|
||||
"outputs": []
|
||||
}
|
||||
```
|
||||
|
||||
Top-level fields:
|
||||
|
||||
- `schema_version`: required. Value `4` for catalog state.
|
||||
- `distributor_version`: optional diagnostic version string.
|
||||
- `created_at`: required RFC3339 timestamp for when the catalog state file was
|
||||
first created.
|
||||
- `updated_at`: required RFC3339 timestamp for the latest catalog state update.
|
||||
- `state.mode`: required. Value `catalog`.
|
||||
- `outputs`: required array of currently managed output records.
|
||||
|
||||
Catalog state should not include top-level `pipeline_id`, `destination_id`,
|
||||
`published_at`, `workflow`, `owners`, `sources`, or a full source manifest.
|
||||
Those concepts belong on output records or in configuration.
|
||||
|
||||
## Output Record Schema
|
||||
|
||||
Each output record should be self-contained enough to support current
|
||||
ownership, pruning, repair, bitrot checks, and basic provenance without a
|
||||
separate owner or source catalog.
|
||||
|
||||
```json
|
||||
{
|
||||
"path": "tomorrow/index.html",
|
||||
"pipeline_id": "weatherreporter.daily",
|
||||
"destination_id": "latest-html",
|
||||
"source": {
|
||||
"id": "weatherreporter.tomorrow",
|
||||
"digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
||||
"created": "2026-06-19T12:00:00Z"
|
||||
},
|
||||
"kind": "generated",
|
||||
"source_path": "report.md",
|
||||
"transform": "markdown_to_html",
|
||||
"sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
|
||||
"size": 12345,
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"url": "https://reports.example.com/weather/tomorrow/"
|
||||
}
|
||||
```
|
||||
|
||||
Required output fields:
|
||||
|
||||
- `path`: destination-relative managed output path.
|
||||
- `pipeline_id`: pipeline that most recently wrote this output path.
|
||||
- `destination_id`: destination that most recently wrote this output path.
|
||||
- `source`: compact identity of the source bundle that produced this output.
|
||||
- `source.id`: source manifest id.
|
||||
- `source.digest`: source manifest digest.
|
||||
- `source.created`: source manifest creation timestamp, RFC3339.
|
||||
- `kind`: `source` or `generated`.
|
||||
- `sha256`: digest of the output bytes.
|
||||
- `size`: output byte size.
|
||||
- `created_at`: RFC3339 timestamp for when this output path first became
|
||||
managed in catalog state.
|
||||
- `updated_at`: RFC3339 timestamp for when this output path was most recently
|
||||
written or updated.
|
||||
|
||||
Optional output fields:
|
||||
|
||||
- `source_path`: present only for generated outputs; source manifest path used
|
||||
to produce the generated output.
|
||||
- `transform`: present only for generated outputs; transform name used to
|
||||
produce the generated output.
|
||||
- `url`: present only when destination link configuration produces a public URL
|
||||
for this output.
|
||||
|
||||
For copied source outputs, `source_path` and `transform` should be omitted. The
|
||||
output `path` is already the copied source artifact path.
|
||||
|
||||
## Useful Properties
|
||||
|
||||
The catalog shape supports both primary workflows without unnecessary
|
||||
normalization:
|
||||
|
||||
- current owner of each path is explicit;
|
||||
- current source identity for each path is explicit;
|
||||
- bitrot checks can compare storage bytes to `sha256`;
|
||||
- pruning can use `updated_at`, optionally scoped by `pipeline_id` and
|
||||
`destination_id`;
|
||||
- repair can remove missing managed output records without consulting an owner
|
||||
or source table;
|
||||
- overwriting an output path updates one output record in place;
|
||||
- no orphaned top-level owner/source records need to be maintained.
|
||||
|
||||
The schema intentionally avoids storing a full source manifest for every output.
|
||||
The source manifest remains the producer-to-distributor validation contract, but
|
||||
catalog state only needs compact source identity for currently managed outputs.
|
||||
|
||||
## Relationship To Existing State And Config Modes
|
||||
|
||||
Existing state modes remain current behavior until catalog mode is implemented:
|
||||
|
||||
- `single_owner` schema version `2` supports one owner for a destination bundle
|
||||
path.
|
||||
- `shared_root` schema version `3` supports multiple owners in one destination
|
||||
root but still keeps owner records with latest source manifests.
|
||||
- `catalog` schema version `4` should support replacement and additive
|
||||
current-state ownership without top-level owner or source catalogs.
|
||||
|
||||
The intended user-facing config should move toward `workflow: replacement` and
|
||||
`workflow: additive` instead of requiring ordinary users to combine
|
||||
`state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict
|
||||
settings.
|
||||
|
||||
Because the project is still alpha pre-release, catalog implementation should be
|
||||
a clean break:
|
||||
|
||||
- remove legacy write paths for current `single_owner` and `shared_root` state;
|
||||
- do not implement migration from schema versions `1`, `2`, or `3`;
|
||||
- do not preserve backwards compatibility for legacy destination state;
|
||||
- when a configured destination writes successfully, write schema version `4`
|
||||
catalog state;
|
||||
- if an existing `.distributor.json` has schema version lower than `4`, treat it
|
||||
as superseded legacy state for planning purposes and overwrite it according to
|
||||
the configured workflow, without attempting conversion.
|
||||
|
||||
For the first catalog run against superseded legacy state:
|
||||
|
||||
- `workflow: replacement` may clear the bounded destination root before writing
|
||||
the planned outputs and schema version `4` catalog state.
|
||||
- `workflow: additive` may overwrite only the planned output paths, then write
|
||||
schema version `4` catalog state containing those planned outputs.
|
||||
- unplanned files left behind by an additive run against superseded legacy state
|
||||
are not recorded in catalog state and are treated as unmanaged content by
|
||||
later catalog runs.
|
||||
|
||||
The target direction is that both primary workflows use catalog state.
|
||||
|
||||
## Prune And Reconcile-State Scope
|
||||
|
||||
Catalog maintenance commands should initially keep the existing ownership
|
||||
selector model:
|
||||
|
||||
- `prune --pipeline <id> --destination <id>` operates only on outputs currently
|
||||
owned by that pipeline/destination.
|
||||
- `reconcile-state --pipeline <id> --destination <id>` repairs only outputs
|
||||
currently owned by that pipeline/destination.
|
||||
- `reconcile-state --all-owners` repairs all output records in the catalog.
|
||||
- `prune` remains scoped to the selected pipeline/destination owner only in the
|
||||
initial catalog implementation.
|
||||
|
||||
Do not add catalog-specific selectors in the initial implementation, such as
|
||||
`--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they
|
||||
are not required to make additive workflow first-class.
|
||||
345
docs/roadmap/implementation.md
Normal file
345
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,345 @@
|
||||
# Catalog State And Workflow Implementation Roadmap
|
||||
|
||||
This is the active staged implementation plan for
|
||||
`docs/roadmap/catalog.md`. The feature roadmap defines the target state model
|
||||
and policy decisions; this document defines the implementation sequence for an
|
||||
LLM coding agent to follow stage by stage.
|
||||
|
||||
Future behavior must remain under `docs/roadmap/` until implemented. Update
|
||||
current-behavior docs only in the stage that implements the corresponding
|
||||
behavior.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
`distributor` currently writes destination `.distributor.json` using
|
||||
single-owner schema version `2` or shared-root schema version `3`. Destination
|
||||
behavior is selected through several low-level knobs: `state.mode`,
|
||||
`reconciliation.mode`, `takeover.mode`, and `transfer`.
|
||||
|
||||
The target behavior is a clean alpha break:
|
||||
|
||||
- all newly written destination state uses schema version `4`;
|
||||
- all destinations use `state.mode: catalog` internally;
|
||||
- user-facing destination behavior is selected by `workflow: additive` or
|
||||
`workflow: replacement`;
|
||||
- `workflow` defaults to `additive`;
|
||||
- legacy destination policy fields are rejected, not aliased;
|
||||
- legacy state schema versions `1`, `2`, and `3` are not migrated.
|
||||
|
||||
## Implementation Principles
|
||||
|
||||
- Keep the source manifest contract unchanged.
|
||||
- Keep backend adapters unaware of catalog and workflow policy.
|
||||
- Treat `workflow` as runtime config, not persisted state.
|
||||
- Keep unmanaged content protected after catalog state exists.
|
||||
- Preserve output `created_at` when a path changes owner; update only
|
||||
`updated_at`.
|
||||
- Prefer removing legacy state/config paths over compatibility shims.
|
||||
- Preserve current public APIs outside destination state/config unless the
|
||||
catalog roadmap explicitly changes them.
|
||||
|
||||
## Active Implementation Stages
|
||||
|
||||
## Stage 1: Destination Workflow Config Clean Break
|
||||
|
||||
Goal: replace low-level destination policy config with the user-facing workflow
|
||||
switch.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add destination `workflow` config with accepted values `additive` and
|
||||
`replacement`.
|
||||
- Default omitted `workflow` to `additive`.
|
||||
- Remove or make unsupported the destination-level config fields `state`,
|
||||
`reconciliation`, `takeover`, and `transfer`.
|
||||
- Ensure configs containing those legacy fields fail clearly. Prefer strict YAML
|
||||
unknown-field failure by removing struct fields; add explicit validation only
|
||||
if clearer errors are needed without weakening strict decoding.
|
||||
- Preserve backend, publish, transform, path mapping, links, retention, and
|
||||
other non-policy destination fields.
|
||||
- Update example configs only when the implementation stage also updates
|
||||
current docs; otherwise keep this stage focused on config code and tests.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/config`
|
||||
- Omitted workflow defaults to `additive`.
|
||||
- `workflow: additive` and `workflow: replacement` validate.
|
||||
- Unknown workflow values fail.
|
||||
- Legacy `state`, `reconciliation`, `takeover`, and `transfer` fields fail.
|
||||
- Existing valid examples are updated or tests are adjusted in the same change if
|
||||
examples currently use legacy fields.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- New configs express destination replacement/additive intent with one field.
|
||||
- No normal config path accepts legacy destination policy knobs.
|
||||
|
||||
## Stage 2: Catalog State Schema Version 4
|
||||
|
||||
Goal: implement the schema version `4` catalog state model in `internal/state`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add catalog state types for top-level schema version `4`, `state.mode:
|
||||
catalog`, and `outputs`.
|
||||
- Add compact per-output source identity with `id`, `digest`, and `created`.
|
||||
- Add catalog output records with required `path`, `pipeline_id`,
|
||||
`destination_id`, `source`, `kind`, `sha256`, `size`, `created_at`, and
|
||||
`updated_at`.
|
||||
- Add optional `source_path`, `transform`, and `url`, present only where allowed
|
||||
by the catalog roadmap.
|
||||
- Validate duplicate paths, invalid owner ids, invalid source identities,
|
||||
invalid output paths, invalid digests, negative sizes, invalid timestamps,
|
||||
invalid kind/transform combinations, and invalid URLs.
|
||||
- Marshal catalog state deterministically with the existing JSON formatting
|
||||
conventions.
|
||||
- Update `ParseDocument` so schema version `4` returns catalog state.
|
||||
- Treat schema versions lower than `4` as superseded legacy state for publish
|
||||
planning, not as readable/migrated active state. Keep enough detection to
|
||||
identify legacy state and avoid treating it as arbitrary invalid JSON.
|
||||
- Reject unsupported future schema versions.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state`
|
||||
- Parse/validate/marshal valid catalog state.
|
||||
- Reject malformed catalog state and invalid output records.
|
||||
- Detect schema versions `1`, `2`, and `3` as superseded legacy state.
|
||||
- Reject future schema versions.
|
||||
- Prove no top-level `owners`, `sources`, workflow, source manifest, pipeline id,
|
||||
destination id, or published timestamp is accepted for catalog state.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- State package has one canonical schema version `4` catalog contract for new
|
||||
writes.
|
||||
- Legacy state is detected but not migrated.
|
||||
|
||||
## Stage 3: Catalog Publish Planning
|
||||
|
||||
Goal: plan publish actions against catalog state and destination workflow.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Replace publish request inputs that consume state/reconciliation/takeover and
|
||||
transfer policy with destination workflow.
|
||||
- Map workflow to catalog actions:
|
||||
- `workflow: additive`: upsert planned outputs and retain all other managed
|
||||
catalog outputs.
|
||||
- `workflow: replacement`: upsert planned outputs and delete catalog outputs
|
||||
owned by the current pipeline/destination that are omitted from the plan.
|
||||
- Planned paths that already exist as catalog-managed outputs may be overwritten
|
||||
and become owned by the current pipeline/destination.
|
||||
- Planned path collisions with storage content not recorded in catalog state
|
||||
fail as unmanaged content.
|
||||
- Matching planned outputs may skip writes when source identity and output digest
|
||||
already match, if that optimization can be implemented without changing
|
||||
externally visible results; otherwise writing idempotently is acceptable.
|
||||
- Existing schema `< 4` state is superseded:
|
||||
- replacement workflow may plan a bounded destination-root clear before
|
||||
writing planned outputs;
|
||||
- additive workflow may plan overwrites for planned paths only and leave
|
||||
unplanned files unmanaged.
|
||||
- Invalid JSON or future schema state remains a conflict, not a superseded
|
||||
legacy state.
|
||||
- Preserve path mapping, publish policy, transforms, links, fixed-path
|
||||
selection, and output collision checks.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/publish`
|
||||
- Additive workflow publishes new catalog state.
|
||||
- Additive workflow overwrites an existing managed output and retains unrelated
|
||||
outputs.
|
||||
- Replacement workflow removes omitted outputs owned by the current
|
||||
pipeline/destination.
|
||||
- Replacement workflow preserves unrelated catalog outputs owned by other
|
||||
pipeline/destination pairs.
|
||||
- Managed output ownership moves to the current pipeline/destination when a
|
||||
planned path is overwritten.
|
||||
- `created_at` is preserved when an existing path changes owner; `updated_at`
|
||||
changes.
|
||||
- Unmanaged path collisions fail.
|
||||
- Legacy schema `< 4` state follows the superseded-state rules above.
|
||||
- Invalid/future state fails.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Publish planning no longer depends on single-owner/shared-root comparison
|
||||
semantics.
|
||||
- Workflow behavior is fully driven by `workflow`.
|
||||
|
||||
## Stage 4: Catalog Publish Execution
|
||||
|
||||
Goal: execute catalog publish plans and write schema version `4` state.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Write only schema version `4` catalog `.distributor.json`.
|
||||
- Implement additive execution as managed upsert of planned outputs plus catalog
|
||||
state update.
|
||||
- Implement replacement execution as managed upsert plus deletion of omitted
|
||||
current-owner catalog outputs.
|
||||
- For replacement over superseded legacy state, clear the bounded destination
|
||||
root before writing planned outputs and catalog state.
|
||||
- For additive over superseded legacy state, overwrite planned paths and write a
|
||||
catalog containing only planned outputs; leave unplanned files unmanaged.
|
||||
- Preserve cleanup behavior on failed writes:
|
||||
- additive cleanup removes outputs newly created by the failed attempt where
|
||||
practical;
|
||||
- replacement cleanup follows existing managed-replacement safety where
|
||||
practical;
|
||||
- state is written only after selected outputs are written.
|
||||
- Preserve link metadata, generated output digests, source output digests,
|
||||
content type behavior, and backend-safe writes.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/publish ./internal/app`
|
||||
- Additive execution writes planned outputs, overwrites managed planned paths,
|
||||
retains unrelated managed outputs, and writes catalog state.
|
||||
- Replacement execution deletes omitted current-owner outputs and preserves
|
||||
unrelated owner outputs.
|
||||
- Superseded legacy replacement clears bounded destination root only.
|
||||
- Superseded legacy additive leaves unplanned files on disk but out of catalog
|
||||
state.
|
||||
- Failed writes do not leave misleading catalog state.
|
||||
- Local, fake-backed SSH, and fake-backed S3 app paths exercise the same publish
|
||||
behavior.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Successful `run` writes only v4 catalog state.
|
||||
- Destination outputs match additive/replacement workflow semantics.
|
||||
|
||||
## Stage 5: Run Reporting, Notifications, And CLI Surface
|
||||
|
||||
Goal: update user-visible run behavior to describe workflow/catalog actions
|
||||
instead of legacy replacement/takeover actions.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Replace legacy action labels tied to state/reconciliation/takeover/transfer
|
||||
with these stable workflow-oriented labels:
|
||||
- `publish_new`
|
||||
- `upsert_additive`
|
||||
- `replace_catalog`
|
||||
- `skip_same`
|
||||
- `force_replace`
|
||||
- `fail_unmanaged`
|
||||
- `fail_conflict`
|
||||
- Ensure text and JSON run summaries include workflow-relevant counters.
|
||||
- Include `workflow` in run action records where useful.
|
||||
- Update fixed-path dry-run warnings to describe additive upsert or replacement
|
||||
clearly.
|
||||
- Ensure notifications use the new action labels.
|
||||
- Remove reporting assumptions that depend on `replace_older`,
|
||||
`replace_newer`, `replace_conflict`, or `replace_takeover`.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- Text dry-run output distinguishes additive from replacement workflow.
|
||||
- JSON output includes workflow and stable action labels.
|
||||
- Summary counters are deterministic.
|
||||
- Notifications fire for additive and replacement writes.
|
||||
- Existing CLI commands still parse and execute with the new config shape.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Operators can understand from dry-run output whether a destination will upsert
|
||||
or replace managed catalog outputs.
|
||||
|
||||
## Stage 6: Catalog Prune And Reconcile-State
|
||||
|
||||
Goal: update maintenance commands to operate on catalog state with the agreed
|
||||
selector model.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Prune:
|
||||
- selected `--pipeline` and `--destination` prune only outputs currently owned
|
||||
by that pipeline/destination;
|
||||
- no `--all-owners` prune mode in the initial catalog implementation;
|
||||
- do not add `--source-id`, `--path-prefix`, or `--kind` selectors.
|
||||
- Reconcile-state:
|
||||
- selected `--pipeline` and `--destination` repair only outputs currently
|
||||
owned by that pipeline/destination;
|
||||
- `--all-owners` repairs all catalog outputs;
|
||||
- unmanaged reporting compares storage entries to all catalog output paths,
|
||||
not just selected owner paths.
|
||||
- Rewrite repaired/pruned state as schema version `4`.
|
||||
- Remove legacy single-owner/shared-root maintenance branches.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/app ./internal/cli`
|
||||
- Prune selects only current-owner catalog outputs.
|
||||
- Prune deletes selected outputs and removes their catalog records.
|
||||
- Reconcile-state selected owner removes only missing outputs for that owner.
|
||||
- `reconcile-state --all-owners` removes missing outputs for all owners.
|
||||
- Unmanaged reporting excludes all catalog-managed paths and reports unrecorded
|
||||
storage entries.
|
||||
- JSON/text output remains stable and clear.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Maintenance commands operate only on catalog state and respect the locked
|
||||
selector rules.
|
||||
|
||||
## Stage 7: Clean Break Removal And Documentation
|
||||
|
||||
Goal: remove legacy state/config behavior and document the implemented catalog
|
||||
workflow model.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Remove dead code for writing schema version `2` single-owner and schema
|
||||
version `3` shared-root state.
|
||||
- Remove legacy config structs/constants/defaults/validation for destination
|
||||
`state`, `reconciliation`, `takeover`, and `transfer` where no longer used.
|
||||
- Remove or rewrite tests that only assert legacy state/config behavior.
|
||||
- Update current-behavior docs:
|
||||
- `docs/config.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/integrations/destination-state.md`
|
||||
- relevant `docs/internal/` docs
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/development.md`
|
||||
- Update examples to use `workflow` and remove legacy fields.
|
||||
- Keep roadmap-only material out of current docs.
|
||||
- Once implemented and documented, remove or rewrite
|
||||
`docs/roadmap/catalog.md` so completed behavior is not described only as
|
||||
future work.
|
||||
|
||||
Tests and checks:
|
||||
|
||||
- `go test ./...`
|
||||
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
|
||||
- `rg -n 'single_owner|shared_root|schema version `2`|schema version `3`' docs internal`
|
||||
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Current docs and examples describe the catalog workflow model.
|
||||
- Legacy destination policy fields and legacy write paths are gone.
|
||||
- Full test suite passes.
|
||||
|
||||
## Refactors To Avoid
|
||||
|
||||
- Do not add top-level `owners` or `sources` catalogs.
|
||||
- Do not persist workflow in `.distributor.json`.
|
||||
- Do not keep deprecated config aliases for legacy destination policy fields.
|
||||
- Do not implement state migration from schema versions `1`, `2`, or `3`.
|
||||
- Do not add catalog-specific prune/reconcile selectors beyond the agreed
|
||||
initial ownership scope.
|
||||
- Do not move catalog policy into storage adapters.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No open questions are known. The catalog roadmap locks the state shape, workflow
|
||||
values, default workflow, clean-break policy, legacy-state behavior, output
|
||||
field selection, `created_at` preservation, and maintenance command scope.
|
||||
@@ -205,7 +205,7 @@ Reference: [Operations](operations.md#forced-replacement-workflow).
|
||||
|
||||
Symptom: `fail_conflict`, `destination source id differs`, `same id and created time but different digest`, `pipeline id ... does not match`, or `destination id ... does not match`.
|
||||
|
||||
Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, or same-created source with different content.
|
||||
Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, shared-root output owner, or same-created source with different content. Valid managed identity, source, and shared-root output-owner mismatches can publish as `replace_takeover` when destination `takeover.mode` allows them, or as `replace_conflict` when `transfer.on_conflict: replace` allows managed conflict replacement.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -214,7 +214,7 @@ cat <destination-path>/.distributor.json
|
||||
go run ./cmd/distributor inspect <source-root>
|
||||
```
|
||||
|
||||
Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. To replace the existing state, configure `transfer.on_conflict: replace`, preview with `--dry-run --force`, then publish with `--force`.
|
||||
Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. For normal managed replacement, configure destination `takeover.mode` to match the intended ownership boundary or configure `transfer.on_conflict: replace`, then preview with `--dry-run`. Use `--force` only for exceptional replacement of unmanaged content or other force-only cases reported as `force_replace`.
|
||||
|
||||
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
|
||||
|
||||
@@ -262,7 +262,7 @@ Reference: [Operations](operations.md#managed-output-pruning).
|
||||
|
||||
## Destination Is Newer Than Source
|
||||
|
||||
Symptom: `skip_destination_newer` or `destination is newer and replacement requires --force`.
|
||||
Symptom: `skip_destination_newer`.
|
||||
|
||||
Likely cause: the destination state records a source manifest with a later `created` timestamp than the current source.
|
||||
|
||||
@@ -272,7 +272,7 @@ Diagnostic:
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run --format json
|
||||
```
|
||||
|
||||
Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run --force`, then publish with `--force`.
|
||||
Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer valid managed state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run`, then publish without `--force`.
|
||||
|
||||
Reference: [Operations](operations.md#forced-replacement-workflow).
|
||||
|
||||
@@ -312,7 +312,7 @@ Reference: [Configuration](config.md#publish-and-transform-policy).
|
||||
|
||||
Symptom: `fail_conflict` with a reason like `destination output path ... is owned by <pipeline>/<destination>`.
|
||||
|
||||
Likely cause: a `state.mode: shared_root` destination planned an output path already recorded in `.distributor.json` for another pipeline/destination owner.
|
||||
Likely cause: a `state.mode: shared_root` destination planned an output path already recorded in `.distributor.json` for another pipeline/destination owner, and `takeover.mode` does not allow that managed path to move to the current owner.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -354,7 +354,7 @@ Diagnostic:
|
||||
find <destination-path> -maxdepth 2 -print
|
||||
```
|
||||
|
||||
Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. In merge mode, previously managed retained or overwritten outputs may remain intentionally. Rerun `--dry-run` before publishing again.
|
||||
Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. In same-source merge mode, previously managed retained or overwritten outputs may remain intentionally. Rerun `--dry-run` before publishing again.
|
||||
|
||||
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
|
||||
|
||||
|
||||
@@ -170,7 +170,7 @@ func (b *Backend) WriteFrom(ctx context.Context, logicalPath string, r io.Reader
|
||||
return storage.Entry{}, storage.NewError(storage.OpWriteFrom, BackendName, logicalPath, storage.ErrConflict, fmt.Errorf("stream size %d does not match expected size %d", written, opts.Size))
|
||||
}
|
||||
if opts.PreferAtomic {
|
||||
if err := b.client.Rename(writePath, nativePath); err != nil {
|
||||
if err := renamePromotedFile(b.client, writePath, nativePath, opts.Overwrite); err != nil {
|
||||
return storage.Entry{}, b.translateError(storage.OpWriteFrom, logicalPath, err)
|
||||
}
|
||||
cleanup = false
|
||||
@@ -178,6 +178,27 @@ func (b *Backend) WriteFrom(ctx context.Context, logicalPath string, r io.Reader
|
||||
return b.Stat(ctx, logicalPath)
|
||||
}
|
||||
|
||||
type sftpRenamer interface {
|
||||
PosixRename(oldname, newname string) error
|
||||
Rename(oldname, newname string) error
|
||||
Remove(path string) error
|
||||
}
|
||||
|
||||
func renamePromotedFile(client sftpRenamer, oldname, newname string, overwrite bool) error {
|
||||
if !overwrite {
|
||||
return client.Rename(oldname, newname)
|
||||
}
|
||||
if err := client.PosixRename(oldname, newname); err == nil {
|
||||
return nil
|
||||
} else if !isReplaceRenameFallbackError(err) {
|
||||
return err
|
||||
}
|
||||
if err := client.Remove(newname); err != nil && !isNotExist(err) {
|
||||
return err
|
||||
}
|
||||
return client.Rename(oldname, newname)
|
||||
}
|
||||
|
||||
func (b *Backend) Stat(ctx context.Context, logicalPath string) (storage.Entry, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return storage.Entry{}, err
|
||||
@@ -464,6 +485,14 @@ func isNotExist(err error) bool {
|
||||
return errors.Is(err, fs.ErrNotExist) || errors.Is(err, os.ErrNotExist) || errors.Is(err, sftp.ErrSSHFxNoSuchFile)
|
||||
}
|
||||
|
||||
func isReplaceRenameFallbackError(err error) bool {
|
||||
if errors.Is(err, sftp.ErrSSHFxFailure) || errors.Is(err, sftp.ErrSSHFxOpUnsupported) {
|
||||
return true
|
||||
}
|
||||
var statusErr *sftp.StatusError
|
||||
return errors.As(err, &statusErr) && (statusErr.FxCode() == sftp.ErrSSHFxFailure || statusErr.FxCode() == sftp.ErrSSHFxOpUnsupported)
|
||||
}
|
||||
|
||||
func (b *Backend) translateError(op, logicalPath string, err error) error {
|
||||
kind := storage.ErrUnknown
|
||||
switch {
|
||||
|
||||
125
internal/adapters/ssh/backend_test.go
Normal file
125
internal/adapters/ssh/backend_test.go
Normal file
@@ -0,0 +1,125 @@
|
||||
package ssh
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/pkg/sftp"
|
||||
)
|
||||
|
||||
func TestRenamePromotedFileUsesPlainRenameWithoutOverwrite(t *testing.T) {
|
||||
client := &recordingRenamer{}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", false); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
if got, want := client.calls, []string{"rename temp index.html"}; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileUsesPosixRenameForOverwrite(t *testing.T) {
|
||||
client := &recordingRenamer{}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
if got, want := client.calls, []string{"posix temp index.html"}; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileFallsBackWhenReplaceRenameUnsupported(t *testing.T) {
|
||||
for _, err := range []error{
|
||||
sftp.ErrSSHFxOpUnsupported,
|
||||
sftp.ErrSSHFxFailure,
|
||||
&sftp.StatusError{Code: uint32(sftp.ErrSSHFxOpUnsupported)},
|
||||
&sftp.StatusError{Code: uint32(sftp.ErrSSHFxFailure)},
|
||||
} {
|
||||
t.Run(err.Error(), func(t *testing.T) {
|
||||
client := &recordingRenamer{posixErr: err}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
want := []string{"posix temp index.html", "remove index.html", "rename temp index.html"}
|
||||
if got := client.calls; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileIgnoresMissingTargetDuringFallback(t *testing.T) {
|
||||
client := &recordingRenamer{
|
||||
posixErr: sftp.ErrSSHFxOpUnsupported,
|
||||
removeErr: &os.PathError{
|
||||
Op: "remove",
|
||||
Path: "index.html",
|
||||
Err: os.ErrNotExist,
|
||||
},
|
||||
}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
want := []string{"posix temp index.html", "remove index.html", "rename temp index.html"}
|
||||
if got := client.calls; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileDoesNotFallbackForPermissionError(t *testing.T) {
|
||||
client := &recordingRenamer{posixErr: sftp.ErrSSHFxPermissionDenied}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); !errors.Is(err, sftp.ErrSSHFxPermissionDenied) {
|
||||
t.Fatalf("renamePromotedFile() error = %v, want permission denied", err)
|
||||
}
|
||||
if got, want := client.calls, []string{"posix temp index.html"}; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileReturnsRemoveFallbackError(t *testing.T) {
|
||||
client := &recordingRenamer{
|
||||
posixErr: sftp.ErrSSHFxOpUnsupported,
|
||||
removeErr: sftp.ErrSSHFxPermissionDenied,
|
||||
}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); !errors.Is(err, sftp.ErrSSHFxPermissionDenied) {
|
||||
t.Fatalf("renamePromotedFile() error = %v, want permission denied", err)
|
||||
}
|
||||
want := []string{"posix temp index.html", "remove index.html"}
|
||||
if got := client.calls; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
type recordingRenamer struct {
|
||||
calls []string
|
||||
posixErr error
|
||||
renameErr error
|
||||
removeErr error
|
||||
}
|
||||
|
||||
func (r *recordingRenamer) PosixRename(oldname, newname string) error {
|
||||
r.calls = append(r.calls, "posix "+oldname+" "+newname)
|
||||
return r.posixErr
|
||||
}
|
||||
|
||||
func (r *recordingRenamer) Rename(oldname, newname string) error {
|
||||
r.calls = append(r.calls, "rename "+oldname+" "+newname)
|
||||
return r.renameErr
|
||||
}
|
||||
|
||||
func (r *recordingRenamer) Remove(path string) error {
|
||||
r.calls = append(r.calls, "remove "+path)
|
||||
return r.removeErr
|
||||
}
|
||||
|
||||
func equalStrings(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for index := range a {
|
||||
if a[index] != b[index] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -211,29 +211,18 @@ func removePrunedStateRecords(ctx context.Context, backend storage.Backend, stat
|
||||
if len(paths) == 0 {
|
||||
return false, nil
|
||||
}
|
||||
if document.SingleOwner != nil {
|
||||
next, changed := state.RemoveMissingOutputs(*document.SingleOwner, paths)
|
||||
if document.Catalog != nil {
|
||||
next, changed := state.RemoveMissingCatalogOwnerOutputs(*document.Catalog, scope, paths)
|
||||
if !changed {
|
||||
return false, nil
|
||||
}
|
||||
next.UpdatedAt = now
|
||||
if err := state.Validate(next); err != nil {
|
||||
if err := state.ValidateCatalog(next); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, writeRepairedState(ctx, backend, statePath, next)
|
||||
}
|
||||
if document.SharedRoot != nil {
|
||||
next, changed := state.RemoveMissingSharedRootOwnerOutputs(*document.SharedRoot, scope, paths)
|
||||
if !changed {
|
||||
return false, nil
|
||||
}
|
||||
next.UpdatedAt = now
|
||||
if err := state.ValidateSharedRoot(next); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, writeRepairedState(ctx, backend, statePath, next)
|
||||
}
|
||||
return false, fmt.Errorf("destination state document is empty")
|
||||
return false, unsupportedStateDocumentError(document)
|
||||
}
|
||||
|
||||
func PlanPrune(document state.StateDocument, policy config.PrunePolicy, options PrunePlanOptions) (PrunePlanReport, error) {
|
||||
@@ -266,17 +255,20 @@ func PlanPrune(document state.StateDocument, policy config.PrunePolicy, options
|
||||
}
|
||||
|
||||
func pruneCandidatesForDocument(document state.StateDocument, scope state.OwnerScope) ([]state.PruneCandidate, error) {
|
||||
if document.SingleOwner != nil {
|
||||
singleOwner := *document.SingleOwner
|
||||
if singleOwner.PipelineID != scope.PipelineID || singleOwner.DestinationID != scope.DestinationID {
|
||||
return nil, fmt.Errorf("state owner is %s/%s, not %s/%s", singleOwner.PipelineID, singleOwner.DestinationID, scope.PipelineID, scope.DestinationID)
|
||||
if document.Catalog != nil {
|
||||
return state.CatalogPruneCandidates(*document.Catalog, scope), nil
|
||||
}
|
||||
return state.SingleOwnerPruneCandidates(singleOwner), nil
|
||||
return nil, unsupportedStateDocumentError(document)
|
||||
}
|
||||
|
||||
func unsupportedStateDocumentError(document state.StateDocument) error {
|
||||
if document.SupersededLegacy != nil {
|
||||
return fmt.Errorf("destination state schema_version %d is superseded legacy state", document.SupersededLegacy.SchemaVersion)
|
||||
}
|
||||
if document.SharedRoot != nil {
|
||||
return state.SharedRootPruneCandidates(*document.SharedRoot, scope), nil
|
||||
if document.SingleOwner != nil || document.SharedRoot != nil {
|
||||
return fmt.Errorf("legacy destination state is not supported by this command")
|
||||
}
|
||||
return nil, fmt.Errorf("destination state document is empty")
|
||||
return fmt.Errorf("destination state document is empty")
|
||||
}
|
||||
|
||||
func pruneOlderThan(policy config.PrunePolicy) *time.Duration {
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
)
|
||||
|
||||
func TestPlanPruneDisabledPolicy(t *testing.T) {
|
||||
document := state.StateDocument{SingleOwner: &state.DistributorState{}}
|
||||
document := state.StateDocument{Catalog: &state.CatalogState{}}
|
||||
report, err := PlanPrune(document, config.PrunePolicy{}, PrunePlanOptions{
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
@@ -28,12 +28,12 @@ func TestPlanPruneDisabledPolicy(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanPruneSingleOwnerOutputs(t *testing.T) {
|
||||
func TestPlanPruneCatalogOutputs(t *testing.T) {
|
||||
now := time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
|
||||
olderThan := config.Duration(48 * time.Hour)
|
||||
destinationState := pruneSingleOwnerState(now)
|
||||
catalog := pruneCatalogState(now)
|
||||
|
||||
report, err := PlanPrune(state.StateDocument{SingleOwner: &destinationState}, config.PrunePolicy{
|
||||
report, err := PlanPrune(state.StateDocument{Catalog: &catalog}, config.PrunePolicy{
|
||||
Enabled: true,
|
||||
OlderThan: &olderThan,
|
||||
}, PrunePlanOptions{
|
||||
@@ -52,12 +52,12 @@ func TestPlanPruneSingleOwnerOutputs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanPruneSharedRootCurrentOwnerOnly(t *testing.T) {
|
||||
func TestPlanPruneCatalogCurrentOwnerOnly(t *testing.T) {
|
||||
now := time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
|
||||
keepLatest := 0
|
||||
sharedRoot := pruneSharedRootState(now)
|
||||
catalog := pruneCatalogState(now)
|
||||
|
||||
report, err := PlanPrune(state.StateDocument{SharedRoot: &sharedRoot}, config.PrunePolicy{
|
||||
report, err := PlanPrune(state.StateDocument{Catalog: &catalog}, config.PrunePolicy{
|
||||
Enabled: true,
|
||||
KeepLatest: &keepLatest,
|
||||
}, PrunePlanOptions{
|
||||
@@ -68,10 +68,10 @@ func TestPlanPruneSharedRootCurrentOwnerOnly(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("PlanPrune() error = %v", err)
|
||||
}
|
||||
if got, want := report.CheckedCount, 1; got != want {
|
||||
if got, want := report.CheckedCount, 2; got != want {
|
||||
t.Fatalf("checked count = %d, want %d", got, want)
|
||||
}
|
||||
if got, want := pruneRecordPaths(report.PrunedOutputs), "archive.txt"; got != want {
|
||||
if got, want := pruneRecordPaths(report.PrunedOutputs), "old.txt,fresh.txt"; got != want {
|
||||
t.Fatalf("pruned = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
@@ -80,8 +80,8 @@ func TestPruneDryRunReportsPlannedDeletesWithoutDeletingOrRewritingState(t *test
|
||||
now := time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
|
||||
backend := fake.New()
|
||||
cfg := pruneS3Config(t, pruneOlderThanPolicy(48*time.Hour))
|
||||
original := pruneSingleOwnerState(now)
|
||||
writeFakeSingleOwnerStateForPrune(t, backend, original)
|
||||
original := pruneCatalogState(now)
|
||||
writeFakeCatalogState(t, backend, original)
|
||||
testutil.WriteFakeFile(t, backend, "unmanaged.txt", "keep")
|
||||
|
||||
report, err := pruneConfigWithBackendFactory(context.Background(), cfg, PruneOptions{
|
||||
@@ -102,12 +102,12 @@ func TestPruneDryRunReportsPlannedDeletesWithoutDeletingOrRewritingState(t *test
|
||||
testutil.AssertFakeFile(t, backend, "old.txt", "managed")
|
||||
testutil.AssertFakeFile(t, backend, "fresh.txt", "managed")
|
||||
testutil.AssertFakeFile(t, backend, "unmanaged.txt", "keep")
|
||||
destinationState := readFakeSingleOwnerState(t, backend)
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "old.txt,fresh.txt" {
|
||||
catalog := readFakeCatalogState(t, backend)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "old.txt,fresh.txt,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want original outputs", got)
|
||||
}
|
||||
if !destinationState.UpdatedAt.Equal(original.UpdatedAt) {
|
||||
t.Fatalf("state updated_at = %s, want original %s", destinationState.UpdatedAt, original.UpdatedAt)
|
||||
if !catalog.UpdatedAt.Equal(original.UpdatedAt) {
|
||||
t.Fatalf("state updated_at = %s, want original %s", catalog.UpdatedAt, original.UpdatedAt)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -115,7 +115,7 @@ func TestPruneApplyDeletesOnlyManagedOutputsAndUpdatesState(t *testing.T) {
|
||||
now := time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
|
||||
backend := fake.New()
|
||||
cfg := pruneS3Config(t, pruneOlderThanPolicy(48*time.Hour))
|
||||
writeFakeSingleOwnerStateForPrune(t, backend, pruneSingleOwnerState(now))
|
||||
writeFakeCatalogState(t, backend, pruneCatalogState(now))
|
||||
testutil.WriteFakeFile(t, backend, "unmanaged.txt", "keep")
|
||||
|
||||
report, err := pruneConfigWithBackendFactory(context.Background(), cfg, PruneOptions{
|
||||
@@ -134,14 +134,18 @@ func TestPruneApplyDeletesOnlyManagedOutputsAndUpdatesState(t *testing.T) {
|
||||
}
|
||||
testutil.AssertFakeMissing(t, backend, "old.txt")
|
||||
testutil.AssertFakeFile(t, backend, "fresh.txt", "managed")
|
||||
testutil.AssertFakeFile(t, backend, "html.txt", "managed")
|
||||
testutil.AssertFakeFile(t, backend, "unmanaged.txt", "keep")
|
||||
assertFakeStateExists(t, backend)
|
||||
destinationState := readFakeSingleOwnerState(t, backend)
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "fresh.txt" {
|
||||
catalog := readFakeCatalogState(t, backend)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "fresh.txt,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want fresh.txt", got)
|
||||
}
|
||||
if !destinationState.UpdatedAt.Equal(now) {
|
||||
t.Fatalf("state updated_at = %s, want %s", destinationState.UpdatedAt, now)
|
||||
if catalog.SchemaVersion != state.CatalogSchemaVersion {
|
||||
t.Fatalf("state schema_version = %d, want %d", catalog.SchemaVersion, state.CatalogSchemaVersion)
|
||||
}
|
||||
if !catalog.UpdatedAt.Equal(now) {
|
||||
t.Fatalf("state updated_at = %s, want %s", catalog.UpdatedAt, now)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -150,7 +154,7 @@ func TestPruneApplyPreservesStateForFailedDeletes(t *testing.T) {
|
||||
backend := fake.New()
|
||||
keepLatest := 0
|
||||
cfg := pruneS3Config(t, config.PrunePolicy{Enabled: true, KeepLatest: &keepLatest})
|
||||
writeFakeSingleOwnerStateForPrune(t, backend, pruneSingleOwnerState(now))
|
||||
writeFakeCatalogState(t, backend, pruneCatalogState(now))
|
||||
failingBackend := failingDeleteBackend{Backend: backend, failPath: "fresh.txt"}
|
||||
|
||||
report, err := pruneConfigWithBackendFactory(context.Background(), cfg, PruneOptions{
|
||||
@@ -170,18 +174,17 @@ func TestPruneApplyPreservesStateForFailedDeletes(t *testing.T) {
|
||||
testutil.AssertFakeMissing(t, backend, "old.txt")
|
||||
testutil.AssertFakeFile(t, backend, "fresh.txt", "managed")
|
||||
assertFakeStateExists(t, backend)
|
||||
destinationState := readFakeSingleOwnerState(t, backend)
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "fresh.txt" {
|
||||
catalog := readFakeCatalogState(t, backend)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "fresh.txt,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want only failed output preserved", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneSharedRootPreservesOtherOwnersWhenScopedToCurrentOwner(t *testing.T) {
|
||||
func TestPrunePreservesOtherOwnersWhenScopedToCurrentOwner(t *testing.T) {
|
||||
now := time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
|
||||
backend := fake.New()
|
||||
keepLatest := 0
|
||||
cfg := pruneS3Config(t, config.PrunePolicy{Enabled: true, KeepLatest: &keepLatest})
|
||||
writeFakeSharedRootStateForApp(t, backend, pruneSharedRootState(now))
|
||||
cfg := pruneS3Config(t, pruneOlderThanPolicy(48*time.Hour))
|
||||
writeFakeCatalogState(t, backend, pruneCatalogState(now))
|
||||
testutil.WriteFakeFile(t, backend, "unmanaged.txt", "keep")
|
||||
|
||||
report, err := pruneConfigWithBackendFactory(context.Background(), cfg, PruneOptions{
|
||||
@@ -192,93 +195,56 @@ func TestPruneSharedRootPreservesOtherOwnersWhenScopedToCurrentOwner(t *testing.
|
||||
if err != nil {
|
||||
t.Fatalf("pruneConfigWithBackendFactory() error = %v", err)
|
||||
}
|
||||
if got, want := pruneRecordPaths(report.DeletedOutputs), "archive.txt"; got != want {
|
||||
if got, want := pruneRecordPaths(report.DeletedOutputs), "old.txt"; got != want {
|
||||
t.Fatalf("deleted outputs = %q, want %q", got, want)
|
||||
}
|
||||
testutil.AssertFakeMissing(t, backend, "archive.txt")
|
||||
testutil.AssertFakeFile(t, backend, "html.txt", "old")
|
||||
testutil.AssertFakeMissing(t, backend, "old.txt")
|
||||
testutil.AssertFakeFile(t, backend, "html.txt", "managed")
|
||||
testutil.AssertFakeFile(t, backend, "unmanaged.txt", "keep")
|
||||
sharedRoot := readFakeSharedRootStateForApp(t, backend)
|
||||
if got := strings.Join(sharedRoot.AllManagedOutputPaths(), ","); got != "html.txt" {
|
||||
t.Fatalf("shared-root outputs = %q, want other owner output preserved", got)
|
||||
catalog := readFakeCatalogState(t, backend)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "fresh.txt,html.txt" {
|
||||
t.Fatalf("catalog outputs = %q, want other owner output preserved", got)
|
||||
}
|
||||
}
|
||||
|
||||
func pruneSingleOwnerState(now time.Time) state.DistributorState {
|
||||
func pruneCatalogState(now time.Time) state.CatalogState {
|
||||
manifest := testutil.ValidManifest(testutil.BundleOptions{})
|
||||
publishedAt := now.Add(-96 * time.Hour)
|
||||
return state.DistributorState{
|
||||
SchemaVersion: state.SchemaVersion,
|
||||
createdAt := now.Add(-96 * time.Hour)
|
||||
source := state.CatalogSourceIdentity{ID: manifest.ID, Digest: manifest.Digest, Created: manifest.Created}
|
||||
return state.CatalogState{
|
||||
SchemaVersion: state.CatalogSchemaVersion,
|
||||
DistributorVersion: "test",
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
State: state.StatePolicy{Mode: state.StateModeCatalog},
|
||||
Outputs: []state.CatalogOutputFile{{
|
||||
Path: "old.txt",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
PublishedAt: publishedAt,
|
||||
CreatedAt: publishedAt,
|
||||
UpdatedAt: publishedAt,
|
||||
State: state.StatePolicy{Mode: state.StateModeSingleOwner},
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: config.ReconciliationModeReplace},
|
||||
Source: state.SourceState{Manifest: manifest},
|
||||
DistributorVersion: "test",
|
||||
Outputs: []state.OutputFile{{
|
||||
Path: "old.txt",
|
||||
Source: source,
|
||||
Kind: state.OutputKindSource,
|
||||
SourcePath: "report.md",
|
||||
SHA256: manifest.Files[0].SHA256,
|
||||
Size: manifest.Files[0].Size,
|
||||
CreatedAt: now.Add(-96 * time.Hour),
|
||||
UpdatedAt: now.Add(-72 * time.Hour),
|
||||
}, {
|
||||
Path: "fresh.txt",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
Source: source,
|
||||
Kind: state.OutputKindSource,
|
||||
SourcePath: "summary.txt",
|
||||
SHA256: manifest.Files[1].SHA256,
|
||||
Size: manifest.Files[1].Size,
|
||||
CreatedAt: now.Add(-24 * time.Hour),
|
||||
UpdatedAt: now.Add(-24 * time.Hour),
|
||||
}},
|
||||
}
|
||||
}
|
||||
|
||||
func pruneSharedRootState(now time.Time) state.SharedRootState {
|
||||
manifest := testutil.ValidManifest(testutil.BundleOptions{})
|
||||
archive := state.CurrentOwnerScope("reports", "archive")
|
||||
html := state.CurrentOwnerScope("reports", "html")
|
||||
return state.SharedRootState{
|
||||
SchemaVersion: state.SharedRootSchemaVersion,
|
||||
DistributorVersion: "test",
|
||||
CreatedAt: now.Add(-96 * time.Hour),
|
||||
UpdatedAt: now.Add(-24 * time.Hour),
|
||||
State: state.StatePolicy{Mode: state.StateModeSharedRoot},
|
||||
Owners: []state.OwnerRecord{{
|
||||
Scope: archive,
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: config.ReconciliationModeReplace},
|
||||
Source: state.SourceState{Manifest: manifest},
|
||||
}, {
|
||||
Scope: html,
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: config.ReconciliationModeReplace},
|
||||
Source: state.SourceState{Manifest: manifest},
|
||||
}},
|
||||
Outputs: []state.SharedRootOutputFile{{
|
||||
Path: "archive.txt",
|
||||
Kind: state.OutputKindSource,
|
||||
SourcePath: "report.md",
|
||||
SHA256: manifest.Files[0].SHA256,
|
||||
Size: manifest.Files[0].Size,
|
||||
Owner: archive,
|
||||
SourceID: manifest.ID,
|
||||
SourceDigest: manifest.Digest,
|
||||
SourceCreated: manifest.Created,
|
||||
CreatedAt: now.Add(-96 * time.Hour),
|
||||
UpdatedAt: now.Add(-72 * time.Hour),
|
||||
CreatedAt: now.Add(-24 * time.Hour),
|
||||
UpdatedAt: now.Add(-24 * time.Hour),
|
||||
}, {
|
||||
Path: "html.txt",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "html",
|
||||
Source: source,
|
||||
Kind: state.OutputKindSource,
|
||||
SourcePath: "summary.txt",
|
||||
SHA256: manifest.Files[1].SHA256,
|
||||
Size: manifest.Files[1].Size,
|
||||
Owner: html,
|
||||
SourceID: manifest.ID,
|
||||
SourceDigest: manifest.Digest,
|
||||
SourceCreated: manifest.Created,
|
||||
CreatedAt: now.Add(-96 * time.Hour),
|
||||
UpdatedAt: now.Add(-72 * time.Hour),
|
||||
}},
|
||||
@@ -319,18 +285,31 @@ func pruneOlderThanPolicy(duration time.Duration) config.PrunePolicy {
|
||||
}
|
||||
}
|
||||
|
||||
func writeFakeSingleOwnerStateForPrune(t *testing.T, backend *fake.Backend, destinationState state.DistributorState) {
|
||||
func writeFakeCatalogState(t *testing.T, backend *fake.Backend, catalog state.CatalogState) {
|
||||
t.Helper()
|
||||
data, err := json.MarshalIndent(destinationState, "", " ")
|
||||
data, err := json.MarshalIndent(catalog, "", " ")
|
||||
if err != nil {
|
||||
t.Fatalf("marshal single-owner state: %v", err)
|
||||
t.Fatalf("marshal catalog state: %v", err)
|
||||
}
|
||||
testutil.WriteFakeFile(t, backend, storage.StateFileName, string(append(data, '\n')))
|
||||
for _, output := range destinationState.Outputs {
|
||||
for _, output := range catalog.Outputs {
|
||||
testutil.WriteFakeFile(t, backend, output.Path, "managed")
|
||||
}
|
||||
}
|
||||
|
||||
func readFakeCatalogState(t *testing.T, backend *fake.Backend) state.CatalogState {
|
||||
t.Helper()
|
||||
data, err := backend.ReadFile(context.Background(), storage.StateFileName)
|
||||
if err != nil {
|
||||
t.Fatalf("read catalog state: %v", err)
|
||||
}
|
||||
catalog, err := state.ParseCatalog(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse catalog state: %v", err)
|
||||
}
|
||||
return catalog
|
||||
}
|
||||
|
||||
func assertFakeStateExists(t *testing.T, backend *fake.Backend) {
|
||||
t.Helper()
|
||||
if _, err := backend.Stat(context.Background(), storage.StateFileName); err != nil {
|
||||
|
||||
@@ -158,62 +158,25 @@ func buildReconcileStateReport(ctx context.Context, backend storage.Backend, pip
|
||||
DryRun: options.DryRun,
|
||||
}
|
||||
scope := state.CurrentOwnerScope(pipeline.ID, destination.ID)
|
||||
if document.SingleOwner != nil {
|
||||
return reconcileSingleOwnerState(ctx, backend, statePath, *document.SingleOwner, scope, report, options)
|
||||
if document.Catalog != nil {
|
||||
return reconcileCatalogState(ctx, backend, statePath, *document.Catalog, scope, report, options)
|
||||
}
|
||||
return reconcileSharedRootState(ctx, backend, statePath, *document.SharedRoot, scope, report, options)
|
||||
return ReconcileStateReport{}, unsupportedStateDocumentError(document)
|
||||
}
|
||||
|
||||
func reconcileSingleOwnerState(ctx context.Context, backend storage.Backend, statePath string, destinationState state.DistributorState, scope state.OwnerScope, report ReconcileStateReport, options ReconcileStateOptions) (ReconcileStateReport, error) {
|
||||
if destinationState.PipelineID != scope.PipelineID || destinationState.DestinationID != scope.DestinationID {
|
||||
return ReconcileStateReport{}, fmt.Errorf("state owner is %s/%s, not %s/%s", destinationState.PipelineID, destinationState.DestinationID, scope.PipelineID, scope.DestinationID)
|
||||
}
|
||||
report.StateSchema = destinationState.SchemaVersion
|
||||
report.OwnerScope = &ReconcileStateOwnerScope{PipelineID: scope.PipelineID, DestinationID: scope.DestinationID}
|
||||
managed := state.ManagedOutputPaths(destinationState)
|
||||
missing, err := missingSingleOwnerOutputs(ctx, backend, destinationState.Outputs)
|
||||
if err != nil {
|
||||
return ReconcileStateReport{}, err
|
||||
}
|
||||
report.CheckedCount = len(managed)
|
||||
report.MissingManagedOutputs = missing
|
||||
unmanaged, err := unmanagedEntries(ctx, backend, managed)
|
||||
if err != nil {
|
||||
return ReconcileStateReport{}, err
|
||||
}
|
||||
report.UnmanagedEntries = unmanaged
|
||||
report.WouldChange = options.DryRun && len(missing) > 0
|
||||
|
||||
if !options.DryRun && len(missing) > 0 {
|
||||
missingPaths := missingReportPaths(missing)
|
||||
next, changed := state.RemoveMissingOutputs(destinationState, missingPaths)
|
||||
report.Changed = changed
|
||||
if changed {
|
||||
next.UpdatedAt = time.Now().UTC()
|
||||
if err := state.Validate(next); err != nil {
|
||||
return ReconcileStateReport{}, err
|
||||
}
|
||||
if err := writeRepairedState(ctx, backend, statePath, next); err != nil {
|
||||
return ReconcileStateReport{}, err
|
||||
}
|
||||
}
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
func reconcileSharedRootState(ctx context.Context, backend storage.Backend, statePath string, sharedRoot state.SharedRootState, scope state.OwnerScope, report ReconcileStateReport, options ReconcileStateOptions) (ReconcileStateReport, error) {
|
||||
report.StateSchema = sharedRoot.SchemaVersion
|
||||
func reconcileCatalogState(ctx context.Context, backend storage.Backend, statePath string, catalog state.CatalogState, scope state.OwnerScope, report ReconcileStateReport, options ReconcileStateOptions) (ReconcileStateReport, error) {
|
||||
report.StateSchema = catalog.SchemaVersion
|
||||
report.OwnerScope = &ReconcileStateOwnerScope{
|
||||
PipelineID: scope.PipelineID,
|
||||
DestinationID: scope.DestinationID,
|
||||
AllOwners: options.AllOwners,
|
||||
}
|
||||
managed := sharedRoot.AllManagedOutputPaths()
|
||||
outputs := sharedRoot.Outputs
|
||||
managed := state.CatalogManagedOutputPaths(catalog)
|
||||
outputs := catalog.Outputs
|
||||
if !options.AllOwners {
|
||||
outputs = sharedRootOutputsForOwner(sharedRoot.Outputs, scope)
|
||||
outputs = state.CatalogOutputsForOwner(catalog.Outputs, scope)
|
||||
}
|
||||
missing, err := missingSharedRootOutputs(ctx, backend, outputs)
|
||||
missing, err := missingCatalogOutputs(ctx, backend, outputs)
|
||||
if err != nil {
|
||||
return ReconcileStateReport{}, err
|
||||
}
|
||||
@@ -228,17 +191,17 @@ func reconcileSharedRootState(ctx context.Context, backend storage.Backend, stat
|
||||
|
||||
if !options.DryRun && len(missing) > 0 {
|
||||
missingPaths := missingReportPaths(missing)
|
||||
var next state.SharedRootState
|
||||
var next state.CatalogState
|
||||
var changed bool
|
||||
if options.AllOwners {
|
||||
next, changed = state.RemoveMissingSharedRootOutputs(sharedRoot, missingPaths)
|
||||
next, changed = state.RemoveMissingCatalogOutputs(catalog, missingPaths)
|
||||
} else {
|
||||
next, changed = state.RemoveMissingSharedRootOwnerOutputs(sharedRoot, scope, missingPaths)
|
||||
next, changed = state.RemoveMissingCatalogOwnerOutputs(catalog, scope, missingPaths)
|
||||
}
|
||||
report.Changed = changed
|
||||
if changed {
|
||||
next.UpdatedAt = time.Now().UTC()
|
||||
if err := state.ValidateSharedRoot(next); err != nil {
|
||||
if err := state.ValidateCatalog(next); err != nil {
|
||||
return ReconcileStateReport{}, err
|
||||
}
|
||||
if err := writeRepairedState(ctx, backend, statePath, next); err != nil {
|
||||
@@ -249,21 +212,7 @@ func reconcileSharedRootState(ctx context.Context, backend storage.Backend, stat
|
||||
return report, nil
|
||||
}
|
||||
|
||||
func missingSingleOwnerOutputs(ctx context.Context, backend storage.Backend, outputs []state.OutputFile) ([]ReconcileStatePath, error) {
|
||||
missing := make([]ReconcileStatePath, 0)
|
||||
for _, output := range outputs {
|
||||
if err := checkManagedOutput(ctx, backend, output.Path); err != nil {
|
||||
if storage.IsNotFound(err) {
|
||||
missing = append(missing, ReconcileStatePath{Path: output.Path, StorageStatus: "missing"})
|
||||
continue
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return missing, nil
|
||||
}
|
||||
|
||||
func missingSharedRootOutputs(ctx context.Context, backend storage.Backend, outputs []state.SharedRootOutputFile) ([]ReconcileStatePath, error) {
|
||||
func missingCatalogOutputs(ctx context.Context, backend storage.Backend, outputs []state.CatalogOutputFile) ([]ReconcileStatePath, error) {
|
||||
missing := make([]ReconcileStatePath, 0)
|
||||
for _, output := range outputs {
|
||||
if err := checkManagedOutput(ctx, backend, output.Path); err != nil {
|
||||
@@ -271,8 +220,8 @@ func missingSharedRootOutputs(ctx context.Context, backend storage.Backend, outp
|
||||
missing = append(missing, ReconcileStatePath{
|
||||
Path: output.Path,
|
||||
OwnerScope: &ReconcileStateOwnerScope{
|
||||
PipelineID: output.Owner.PipelineID,
|
||||
DestinationID: output.Owner.DestinationID,
|
||||
PipelineID: output.PipelineID,
|
||||
DestinationID: output.DestinationID,
|
||||
},
|
||||
StorageStatus: "missing",
|
||||
})
|
||||
@@ -320,16 +269,6 @@ func unmanagedEntries(ctx context.Context, backend storage.Backend, managedPaths
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
func sharedRootOutputsForOwner(outputs []state.SharedRootOutputFile, scope state.OwnerScope) []state.SharedRootOutputFile {
|
||||
selected := make([]state.SharedRootOutputFile, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
if output.Owner == scope {
|
||||
selected = append(selected, output)
|
||||
}
|
||||
}
|
||||
return selected
|
||||
}
|
||||
|
||||
func missingReportPaths(missing []ReconcileStatePath) []string {
|
||||
paths := make([]string, 0, len(missing))
|
||||
for _, item := range missing {
|
||||
|
||||
@@ -2,7 +2,6 @@ package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -17,9 +16,9 @@ import (
|
||||
func TestReconcileStateDryRunReportsMissingManagedOutputsWithoutRewrite(t *testing.T) {
|
||||
backend := fake.New()
|
||||
cfg := reconcileStateS3Config(t)
|
||||
manifest := testutil.ValidManifest(testutil.BundleOptions{})
|
||||
testutil.WriteFakeDestinationState(t, backend, "", manifest, testutil.DestinationStateOptions{})
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"summary.txt"}, storage.DeleteOptions{}); err != nil {
|
||||
catalog := pruneCatalogState(time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC))
|
||||
writeFakeCatalogState(t, backend, catalog)
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"fresh.txt"}, storage.DeleteOptions{}); err != nil {
|
||||
t.Fatalf("delete managed output: %v", err)
|
||||
}
|
||||
testutil.WriteFakeFile(t, backend, "extra.txt", "unmanaged")
|
||||
@@ -35,14 +34,14 @@ func TestReconcileStateDryRunReportsMissingManagedOutputsWithoutRewrite(t *testi
|
||||
if !report.WouldChange || report.Changed {
|
||||
t.Fatalf("report changed=%t would_change=%t, want dry-run pending change", report.Changed, report.WouldChange)
|
||||
}
|
||||
if got := reportPathList(report.MissingManagedOutputs); got != "summary.txt" {
|
||||
t.Fatalf("missing outputs = %q, want summary.txt", got)
|
||||
if got := reportPathList(report.MissingManagedOutputs); got != "fresh.txt" {
|
||||
t.Fatalf("missing outputs = %q, want fresh.txt", got)
|
||||
}
|
||||
if got := entryPathList(report.UnmanagedEntries); got != "extra.txt" {
|
||||
t.Fatalf("unmanaged entries = %q, want extra.txt", got)
|
||||
}
|
||||
destinationState := readFakeSingleOwnerState(t, backend)
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "report.md,summary.txt" {
|
||||
repaired := readFakeCatalogState(t, backend)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(repaired), ","); got != "old.txt,fresh.txt,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want original outputs", got)
|
||||
}
|
||||
}
|
||||
@@ -50,9 +49,8 @@ func TestReconcileStateDryRunReportsMissingManagedOutputsWithoutRewrite(t *testi
|
||||
func TestReconcileStateApplyRemovesMissingRecordsAndPreservesUnmanagedFiles(t *testing.T) {
|
||||
backend := fake.New()
|
||||
cfg := reconcileStateS3Config(t)
|
||||
manifest := testutil.ValidManifest(testutil.BundleOptions{})
|
||||
testutil.WriteFakeDestinationState(t, backend, "", manifest, testutil.DestinationStateOptions{})
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"summary.txt"}, storage.DeleteOptions{}); err != nil {
|
||||
writeFakeCatalogState(t, backend, pruneCatalogState(time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)))
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"fresh.txt"}, storage.DeleteOptions{}); err != nil {
|
||||
t.Fatalf("delete managed output: %v", err)
|
||||
}
|
||||
testutil.WriteFakeFile(t, backend, "extra.txt", "unmanaged")
|
||||
@@ -67,12 +65,15 @@ func TestReconcileStateApplyRemovesMissingRecordsAndPreservesUnmanagedFiles(t *t
|
||||
if !report.Changed || report.WouldChange {
|
||||
t.Fatalf("report changed=%t would_change=%t, want applied change", report.Changed, report.WouldChange)
|
||||
}
|
||||
destinationState := readFakeSingleOwnerState(t, backend)
|
||||
if err := state.Validate(destinationState); err != nil {
|
||||
t.Fatalf("Validate() repaired state error = %v", err)
|
||||
repaired := readFakeCatalogState(t, backend)
|
||||
if err := state.ValidateCatalog(repaired); err != nil {
|
||||
t.Fatalf("ValidateCatalog() repaired state error = %v", err)
|
||||
}
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "report.md" {
|
||||
t.Fatalf("state outputs = %q, want report.md", got)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(repaired), ","); got != "old.txt,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want old.txt,html.txt", got)
|
||||
}
|
||||
if repaired.SchemaVersion != state.CatalogSchemaVersion {
|
||||
t.Fatalf("state schema_version = %d, want %d", repaired.SchemaVersion, state.CatalogSchemaVersion)
|
||||
}
|
||||
testutil.AssertFakeFile(t, backend, "extra.txt", "unmanaged")
|
||||
}
|
||||
@@ -99,12 +100,11 @@ func TestReconcileStateInvalidStateFailsWithoutRewrite(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReconcileStateSharedRootOwnerScopeRepairsCurrentOwnerOnly(t *testing.T) {
|
||||
func TestReconcileStateOwnerScopeRepairsCurrentOwnerOnly(t *testing.T) {
|
||||
backend := fake.New()
|
||||
cfg := reconcileStateS3Config(t)
|
||||
sharedRoot := reconcileSharedRootFixture(t)
|
||||
writeFakeSharedRootStateForApp(t, backend, sharedRoot)
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"report.md", "report.html"}, storage.DeleteOptions{}); err != nil {
|
||||
writeFakeCatalogState(t, backend, pruneCatalogState(time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)))
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"old.txt", "html.txt"}, storage.DeleteOptions{}); err != nil {
|
||||
t.Fatalf("delete managed outputs: %v", err)
|
||||
}
|
||||
|
||||
@@ -118,18 +118,17 @@ func TestReconcileStateSharedRootOwnerScopeRepairsCurrentOwnerOnly(t *testing.T)
|
||||
if !report.Changed {
|
||||
t.Fatal("report changed = false, want true")
|
||||
}
|
||||
repaired := readFakeSharedRootStateForApp(t, backend)
|
||||
if got := strings.Join(repaired.AllManagedOutputPaths(), ","); got != "report.html" {
|
||||
t.Fatalf("shared-root outputs = %q, want other owner output preserved", got)
|
||||
repaired := readFakeCatalogState(t, backend)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(repaired), ","); got != "fresh.txt,html.txt" {
|
||||
t.Fatalf("catalog outputs = %q, want other owner output preserved", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReconcileStateSharedRootAllOwnersRepairsEveryOwner(t *testing.T) {
|
||||
func TestReconcileStateAllOwnersRepairsEveryOwner(t *testing.T) {
|
||||
backend := fake.New()
|
||||
cfg := reconcileStateS3Config(t)
|
||||
sharedRoot := reconcileSharedRootFixture(t)
|
||||
writeFakeSharedRootStateForApp(t, backend, sharedRoot)
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"report.md", "report.html"}, storage.DeleteOptions{}); err != nil {
|
||||
writeFakeCatalogState(t, backend, pruneCatalogState(time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)))
|
||||
if err := backend.DeleteManagedOutputs(context.Background(), "", []string{"old.txt", "html.txt"}, storage.DeleteOptions{}); err != nil {
|
||||
t.Fatalf("delete managed outputs: %v", err)
|
||||
}
|
||||
|
||||
@@ -141,12 +140,12 @@ func TestReconcileStateSharedRootAllOwnersRepairsEveryOwner(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("reconcileStateConfigWithBackendFactory() error = %v", err)
|
||||
}
|
||||
if !report.Changed || report.CheckedCount != 2 {
|
||||
if !report.Changed || report.CheckedCount != 3 {
|
||||
t.Fatalf("report changed=%t checked=%d, want all-owner repair", report.Changed, report.CheckedCount)
|
||||
}
|
||||
repaired := readFakeSharedRootStateForApp(t, backend)
|
||||
if got := repaired.AllManagedOutputPaths(); len(got) != 0 {
|
||||
t.Fatalf("shared-root outputs = %#v, want none", got)
|
||||
repaired := readFakeCatalogState(t, backend)
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(repaired), ","); got != "fresh.txt" {
|
||||
t.Fatalf("catalog outputs = %q, want fresh.txt", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -165,93 +164,6 @@ func reconcileStateS3Config(t *testing.T) config.Config {
|
||||
return cfg
|
||||
}
|
||||
|
||||
func readFakeSingleOwnerState(t *testing.T, backend *fake.Backend) state.DistributorState {
|
||||
t.Helper()
|
||||
data, err := backend.ReadFile(context.Background(), storage.StateFileName)
|
||||
if err != nil {
|
||||
t.Fatalf("read state: %v", err)
|
||||
}
|
||||
destinationState, err := state.Parse(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse state: %v", err)
|
||||
}
|
||||
return destinationState
|
||||
}
|
||||
|
||||
func writeFakeSharedRootStateForApp(t *testing.T, backend *fake.Backend, sharedRoot state.SharedRootState) {
|
||||
t.Helper()
|
||||
data, err := json.MarshalIndent(sharedRoot, "", " ")
|
||||
if err != nil {
|
||||
t.Fatalf("marshal shared-root state: %v", err)
|
||||
}
|
||||
testutil.WriteFakeFile(t, backend, storage.StateFileName, string(append(data, '\n')))
|
||||
for _, output := range sharedRoot.Outputs {
|
||||
testutil.WriteFakeFile(t, backend, output.Path, "old")
|
||||
}
|
||||
}
|
||||
|
||||
func readFakeSharedRootStateForApp(t *testing.T, backend *fake.Backend) state.SharedRootState {
|
||||
t.Helper()
|
||||
data, err := backend.ReadFile(context.Background(), storage.StateFileName)
|
||||
if err != nil {
|
||||
t.Fatalf("read shared-root state: %v", err)
|
||||
}
|
||||
sharedRoot, err := state.ParseSharedRoot(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse shared-root state: %v", err)
|
||||
}
|
||||
return sharedRoot
|
||||
}
|
||||
|
||||
func reconcileSharedRootFixture(t *testing.T) state.SharedRootState {
|
||||
t.Helper()
|
||||
source := testutil.ValidManifest(testutil.BundleOptions{})
|
||||
htmlSource := source
|
||||
createdAt := time.Date(2026, 5, 30, 11, 12, 0, 0, time.UTC)
|
||||
return state.SharedRootState{
|
||||
SchemaVersion: state.SharedRootSchemaVersion,
|
||||
DistributorVersion: "test",
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
State: state.StatePolicy{Mode: state.StateModeSharedRoot},
|
||||
Owners: []state.OwnerRecord{{
|
||||
Scope: state.CurrentOwnerScope("reports", "archive"),
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: config.ReconciliationModeReplace},
|
||||
Source: state.SourceState{Manifest: source},
|
||||
}, {
|
||||
Scope: state.CurrentOwnerScope("reports", "html"),
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: config.ReconciliationModeMerge},
|
||||
Source: state.SourceState{Manifest: htmlSource},
|
||||
}},
|
||||
Outputs: []state.SharedRootOutputFile{{
|
||||
Path: "report.md",
|
||||
Kind: state.OutputKindSource,
|
||||
SourcePath: "report.md",
|
||||
SHA256: source.Files[0].SHA256,
|
||||
Size: source.Files[0].Size,
|
||||
Owner: state.CurrentOwnerScope("reports", "archive"),
|
||||
SourceID: source.ID,
|
||||
SourceDigest: source.Digest,
|
||||
SourceCreated: source.Created,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
}, {
|
||||
Path: "report.html",
|
||||
Kind: state.OutputKindGenerated,
|
||||
SourcePath: "report.md",
|
||||
Transform: "markdown_to_html",
|
||||
SHA256: "sha256:" + strings.Repeat("a", 64),
|
||||
Size: 128,
|
||||
Owner: state.CurrentOwnerScope("reports", "html"),
|
||||
SourceID: htmlSource.ID,
|
||||
SourceDigest: htmlSource.Digest,
|
||||
SourceCreated: htmlSource.Created,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
}},
|
||||
}
|
||||
}
|
||||
|
||||
func reportPathList(paths []ReconcileStatePath) string {
|
||||
values := make([]string, 0, len(paths))
|
||||
for _, path := range paths {
|
||||
|
||||
@@ -69,10 +69,8 @@ func processDestinationSelection(ctx context.Context, request runDestinationRequ
|
||||
Publish: *request.destination.Publish,
|
||||
Transform: request.destination.Transform,
|
||||
Links: request.destination.Links,
|
||||
State: request.destination.State,
|
||||
Reconciliation: request.destination.Reconciliation,
|
||||
Workflow: request.destination.Workflow,
|
||||
Transformers: request.transforms,
|
||||
Transfer: request.destination.Transfer,
|
||||
DistributorVersion: Version,
|
||||
Force: request.options.Force,
|
||||
}
|
||||
@@ -82,8 +80,8 @@ func processDestinationSelection(ctx context.Context, request runDestinationRequ
|
||||
}
|
||||
if isFixedPathDestination(request.destination) {
|
||||
plan.PathMapping = config.PathMappingFixed
|
||||
if request.options.DryRun && isDestructiveFixedPathAction(plan.Action) {
|
||||
warning := fixedPathReplacementWarning(plan)
|
||||
if request.options.DryRun && isFixedPathWorkflowAction(plan.Action) {
|
||||
warning := fixedPathWorkflowWarning(plan)
|
||||
request.recorder.addPipelineWarning(request.pipelineIndex, warning)
|
||||
}
|
||||
}
|
||||
@@ -141,6 +139,7 @@ func (recorder *runReportRecorder) recordDestinationFailure(pipelineIndex int, f
|
||||
recorder.failures.add(failure.pipelineID, failure.destinationID, failure.backend, storage.DisplayPath(failure.bundlePath), failure.err)
|
||||
recorder.summary.recordFailure()
|
||||
if includeAction {
|
||||
recorder.summary.recordFailureAction(action.Action)
|
||||
recorder.addPipelineAction(pipelineIndex, action)
|
||||
}
|
||||
}
|
||||
@@ -161,5 +160,8 @@ func completePlanIdentity(plan publish.Plan, pipeline config.Pipeline, destinati
|
||||
if plan.DestinationBundlePath == "" {
|
||||
plan.DestinationBundlePath = selection.DestinationBundlePath
|
||||
}
|
||||
if plan.Workflow == "" {
|
||||
plan.Workflow = destination.Workflow
|
||||
}
|
||||
return plan
|
||||
}
|
||||
|
||||
@@ -6,7 +6,7 @@ import (
|
||||
)
|
||||
|
||||
func shouldNotify(action publish.Action) bool {
|
||||
return action == publish.ActionPublishNew || action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
|
||||
return action == publish.ActionPublishNew || action == publish.ActionUpsertAdditive || action == publish.ActionReplaceCatalog || action == publish.ActionForceReplace
|
||||
}
|
||||
|
||||
func notifyEvent(plan publish.Plan) notify.Event {
|
||||
|
||||
@@ -60,7 +60,7 @@ func writeRunActionLine(w io.Writer, action RunActionRecord) {
|
||||
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=error reason=%q\n", action.BundlePath, destinationID, action.Backend, pathMappingRecordSummary(action), action.Reason)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=%s outputs=%s reason=%q\n", action.BundlePath, action.DestinationID, action.Backend, pathMappingRecordSummary(action), action.Action, outputRecordSummary(action.Outputs), action.Reason)
|
||||
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=%s%s outputs=%s reason=%q\n", action.BundlePath, action.DestinationID, action.Backend, pathMappingRecordSummary(action), action.Action, workflowRecordSummary(action), outputRecordSummary(action.Outputs), action.Reason)
|
||||
}
|
||||
|
||||
func pathMappingRecordSummary(action RunActionRecord) string {
|
||||
@@ -70,6 +70,13 @@ func pathMappingRecordSummary(action RunActionRecord) string {
|
||||
return fmt.Sprintf(" path_mapping=fixed target=%s", action.DestinationPath)
|
||||
}
|
||||
|
||||
func workflowRecordSummary(action RunActionRecord) string {
|
||||
if action.Workflow == "" {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf(" workflow=%s", action.Workflow)
|
||||
}
|
||||
|
||||
func outputRecordSummary(outputs []RunOutputRecord) string {
|
||||
if len(outputs) == 0 {
|
||||
return "none"
|
||||
@@ -136,6 +143,7 @@ type RunActionRecord struct {
|
||||
BundlePath string `json:"bundle_path"`
|
||||
DestinationPath string `json:"destination_path"`
|
||||
PathMapping string `json:"path_mapping,omitempty"`
|
||||
Workflow string `json:"workflow,omitempty"`
|
||||
Action string `json:"action"`
|
||||
PrimaryURL string `json:"primary_url,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
@@ -158,6 +166,13 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) RunActi
|
||||
if destinationID == "" {
|
||||
destinationID = "unknown"
|
||||
}
|
||||
action := "error"
|
||||
outputs := []RunOutputRecord{}
|
||||
switch plan.Action {
|
||||
case publish.ActionFailUnmanaged, publish.ActionFailConflict:
|
||||
action = string(plan.Action)
|
||||
outputs = runOutputsFromPlan(plan.Outputs)
|
||||
}
|
||||
return RunActionRecord{
|
||||
PipelineID: plan.PipelineID,
|
||||
DestinationID: destinationID,
|
||||
@@ -166,10 +181,11 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) RunActi
|
||||
BundlePath: storage.DisplayPath(plan.BundlePath),
|
||||
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
||||
PathMapping: plan.PathMapping,
|
||||
Action: "error",
|
||||
Workflow: plan.Workflow,
|
||||
Action: action,
|
||||
PrimaryURL: plan.PrimaryURL,
|
||||
Reason: planErr.Error(),
|
||||
Outputs: []RunOutputRecord{},
|
||||
Outputs: outputs,
|
||||
}
|
||||
}
|
||||
return RunActionRecord{
|
||||
@@ -180,6 +196,7 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) RunActi
|
||||
BundlePath: storage.DisplayPath(plan.BundlePath),
|
||||
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
||||
PathMapping: plan.PathMapping,
|
||||
Workflow: plan.Workflow,
|
||||
Action: string(plan.Action),
|
||||
PrimaryURL: plan.PrimaryURL,
|
||||
Reason: plan.Reason,
|
||||
|
||||
@@ -62,12 +62,22 @@ func fixedPathSelectionWarning(pipelineID, destinationID string, selections []de
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed candidates=%d selected_bundle=%s destination_bundle=.", pipelineID, destinationID, candidateCount, selected)}
|
||||
}
|
||||
|
||||
func isDestructiveFixedPathAction(action publish.Action) bool {
|
||||
return action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
|
||||
func isFixedPathWorkflowAction(action publish.Action) bool {
|
||||
return action == publish.ActionUpsertAdditive || action == publish.ActionReplaceCatalog || action == publish.ActionForceReplace
|
||||
}
|
||||
|
||||
func fixedPathReplacementWarning(plan publish.Plan) OutputWarning {
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed action=%s replaces destination root for selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Action, storage.DisplayPath(plan.BundlePath))}
|
||||
func fixedPathWorkflowWarning(plan publish.Plan) OutputWarning {
|
||||
switch plan.Action {
|
||||
case publish.ActionReplaceCatalog:
|
||||
if plan.ClearDestinationRoot {
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed workflow=%s action=%s clears destination root before writing selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Workflow, plan.Action, storage.DisplayPath(plan.BundlePath))}
|
||||
}
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed workflow=%s action=%s replaces current-owner catalog outputs for selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Workflow, plan.Action, storage.DisplayPath(plan.BundlePath))}
|
||||
case publish.ActionUpsertAdditive:
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed workflow=%s action=%s upserts planned outputs at destination root for selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Workflow, plan.Action, storage.DisplayPath(plan.BundlePath))}
|
||||
default:
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed workflow=%s action=%s writes selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Workflow, plan.Action, storage.DisplayPath(plan.BundlePath))}
|
||||
}
|
||||
}
|
||||
|
||||
func destinationIDs(destinations []config.Destination) []string {
|
||||
|
||||
@@ -10,9 +10,12 @@ type runSummary struct {
|
||||
dryRun bool
|
||||
planned int
|
||||
publishNew int
|
||||
replaceOlder int
|
||||
upsertAdditive int
|
||||
replaceCatalog int
|
||||
skipSame int
|
||||
forceReplace int
|
||||
skipped int
|
||||
failUnmanaged int
|
||||
failConflict int
|
||||
failures int
|
||||
fixedPath int
|
||||
}
|
||||
@@ -22,12 +25,23 @@ func (s *runSummary) recordPlan(action publish.Action) {
|
||||
switch action {
|
||||
case publish.ActionPublishNew:
|
||||
s.publishNew++
|
||||
case publish.ActionReplaceOlder:
|
||||
s.replaceOlder++
|
||||
case publish.ActionUpsertAdditive:
|
||||
s.upsertAdditive++
|
||||
case publish.ActionReplaceCatalog:
|
||||
s.replaceCatalog++
|
||||
case publish.ActionForceReplace:
|
||||
s.forceReplace++
|
||||
case publish.ActionSkipSame, publish.ActionSkipDestinationNewer:
|
||||
s.skipped++
|
||||
s.skipSame++
|
||||
}
|
||||
}
|
||||
|
||||
func (s *runSummary) recordFailureAction(action string) {
|
||||
switch action {
|
||||
case string(publish.ActionFailUnmanaged):
|
||||
s.failUnmanaged++
|
||||
case string(publish.ActionFailConflict):
|
||||
s.failConflict++
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,16 +57,19 @@ type RunSummaryCounters struct {
|
||||
Status string `json:"status"`
|
||||
Planned int `json:"planned"`
|
||||
PublishNew int `json:"publish_new"`
|
||||
ReplaceOlder int `json:"replace_older"`
|
||||
UpsertAdditive int `json:"upsert_additive"`
|
||||
ReplaceCatalog int `json:"replace_catalog"`
|
||||
SkipSame int `json:"skip_same"`
|
||||
ForceReplace int `json:"force_replace"`
|
||||
Skipped int `json:"skipped"`
|
||||
FailUnmanaged int `json:"fail_unmanaged"`
|
||||
FailConflict int `json:"fail_conflict"`
|
||||
Failed int `json:"failed"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
FixedPath int `json:"fixed_path"`
|
||||
}
|
||||
|
||||
func (s RunSummaryCounters) Line() string {
|
||||
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d replace_older=%d force_replace=%d skipped=%d failed=%d dry_run=%t fixed_path=%d", s.Status, s.Planned, s.PublishNew, s.ReplaceOlder, s.ForceReplace, s.Skipped, s.Failed, s.DryRun, s.FixedPath)
|
||||
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d upsert_additive=%d replace_catalog=%d skip_same=%d force_replace=%d fail_unmanaged=%d fail_conflict=%d failed=%d dry_run=%t fixed_path=%d", s.Status, s.Planned, s.PublishNew, s.UpsertAdditive, s.ReplaceCatalog, s.SkipSame, s.ForceReplace, s.FailUnmanaged, s.FailConflict, s.Failed, s.DryRun, s.FixedPath)
|
||||
}
|
||||
|
||||
func (s runSummary) Result() RunSummaryCounters {
|
||||
@@ -64,9 +81,12 @@ func (s runSummary) Result() RunSummaryCounters {
|
||||
Status: status,
|
||||
Planned: s.planned,
|
||||
PublishNew: s.publishNew,
|
||||
ReplaceOlder: s.replaceOlder,
|
||||
UpsertAdditive: s.upsertAdditive,
|
||||
ReplaceCatalog: s.replaceCatalog,
|
||||
SkipSame: s.skipSame,
|
||||
ForceReplace: s.forceReplace,
|
||||
Skipped: s.skipped,
|
||||
FailUnmanaged: s.failUnmanaged,
|
||||
FailConflict: s.failConflict,
|
||||
Failed: s.failures,
|
||||
DryRun: s.dryRun,
|
||||
FixedPath: s.fixedPath,
|
||||
|
||||
@@ -15,7 +15,6 @@ import (
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/notify"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/publish"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
|
||||
@@ -42,8 +41,8 @@ func TestRunDryRunPrintsConfigSummary(t *testing.T) {
|
||||
for _, want := range []string{
|
||||
"Configured pipelines: 1",
|
||||
"- pipeline=reports source=local bundles=1 destinations=archive",
|
||||
"bundle=. destination=archive backend=local action=publish_new outputs=report.md,summary.txt",
|
||||
"Final status: ok planned=1 publish_new=1 replace_older=0 force_replace=0 skipped=0 failed=0 dry_run=true",
|
||||
"bundle=. destination=archive backend=local action=publish_new workflow=additive outputs=report.md,summary.txt",
|
||||
"Final status: ok planned=1 publish_new=1 upsert_additive=0 replace_catalog=0 skip_same=0 force_replace=0 fail_unmanaged=0 fail_conflict=0 failed=0 dry_run=true",
|
||||
} {
|
||||
if !strings.Contains(output, want) {
|
||||
t.Fatalf("Run() output = %q, want substring %q", output, want)
|
||||
@@ -222,67 +221,6 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunSharedRootDryRunWritesNoOutputsOrState(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{
|
||||
Files: []testFile{{Path: "report.md", Data: "# Report\n"}},
|
||||
})
|
||||
configPath := writeSharedRootLocalConfig(t, sourceRoot, destinationRoot)
|
||||
|
||||
cfg, err := config.LoadFile(configPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load config: %v", err)
|
||||
}
|
||||
report, err := buildRunReportWithBackendFactory(context.Background(), cfg, RunOptions{DryRun: true}, newBackendFactoryWithEnvironment)
|
||||
if err != nil {
|
||||
t.Fatalf("dry-run buildRunReportWithBackendFactory() error = %v", err)
|
||||
}
|
||||
if got, want := report.Actions[0].Action, string(publish.ActionPublishNew); got != want {
|
||||
t.Fatalf("dry-run action = %q, want %q", got, want)
|
||||
}
|
||||
if _, statErr := os.Stat(filepath.Join(destinationRoot, "report.md")); !os.IsNotExist(statErr) {
|
||||
t.Fatalf("output stat error = %v, want absent", statErr)
|
||||
}
|
||||
if _, statErr := os.Stat(filepath.Join(destinationRoot, storage.StateFileName)); !os.IsNotExist(statErr) {
|
||||
t.Fatalf("state file stat error = %v, want absent", statErr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunPublishesTwoPipelinesIntoSharedRoot(t *testing.T) {
|
||||
firstSourceRoot := t.TempDir()
|
||||
secondSourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, firstSourceRoot, "", testBundleOptions{
|
||||
ID: "reports.first",
|
||||
Files: []testFile{{Path: "first.md", Data: "# First\n"}},
|
||||
})
|
||||
writeSourceBundle(t, secondSourceRoot, "", testBundleOptions{
|
||||
ID: "reports.second",
|
||||
Files: []testFile{{Path: "second.md", Data: "# Second\n"}},
|
||||
})
|
||||
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeTwoPipelineSharedRootConfig(t, firstSourceRoot, secondSourceRoot, destinationRoot)})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "first.md"), "# First\n")
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "second.md"), "# Second\n")
|
||||
destinationState := readSharedRootStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got, want := len(destinationState.Owners), 2; got != want {
|
||||
t.Fatalf("owner count = %d, want %d", got, want)
|
||||
}
|
||||
if _, ok := destinationState.Owner(state.CurrentOwnerScope("reports-first", "archive")); !ok {
|
||||
t.Fatal("reports-first/archive owner missing")
|
||||
}
|
||||
if _, ok := destinationState.Owner(state.CurrentOwnerScope("reports-second", "archive")); !ok {
|
||||
t.Fatal("reports-second/archive owner missing")
|
||||
}
|
||||
if got, want := strings.Join(destinationState.AllManagedOutputPaths(), ","), "first.md,second.md"; got != want {
|
||||
t.Fatalf("managed paths = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunPipelineWithLocalSourcePublishesConfiguredDestination(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
@@ -568,7 +506,7 @@ func TestRunFixedPathDryRunWarnsForReplacement(t *testing.T) {
|
||||
{Path: "summary.txt", Data: "Old summary\n"},
|
||||
},
|
||||
})
|
||||
configPath := testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
|
||||
configPath := writeLocalConfigWithWorkflow(t, sourceRoot, destinationRoot, config.PathMappingFixed, config.WorkflowReplacement)
|
||||
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
|
||||
t.Fatalf("first Run() error = %v", err)
|
||||
}
|
||||
@@ -592,8 +530,9 @@ func TestRunFixedPathDryRunWarnsForReplacement(t *testing.T) {
|
||||
}
|
||||
output := stdout.String()
|
||||
for _, want := range []string{
|
||||
"Warning: pipeline=reports destination=archive path_mapping=fixed action=replace_older replaces destination root for selected_bundle=new",
|
||||
"bundle=new destination=archive backend=local path_mapping=fixed target=. action=replace_older",
|
||||
"Warning: pipeline=reports destination=archive path_mapping=fixed workflow=replacement action=replace_catalog replaces current-owner catalog outputs for selected_bundle=new",
|
||||
"bundle=new destination=archive backend=local path_mapping=fixed target=. action=replace_catalog workflow=replacement outputs=report.md,summary.txt reason=\"\"",
|
||||
"replace_catalog=1",
|
||||
} {
|
||||
if !strings.Contains(output, want) {
|
||||
t.Fatalf("stdout = %q, want substring %q", output, want)
|
||||
@@ -602,7 +541,7 @@ func TestRunFixedPathDryRunWarnsForReplacement(t *testing.T) {
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
|
||||
}
|
||||
|
||||
func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
|
||||
func TestRunJSONIncludesWorkflowActionAndSummary(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{
|
||||
@@ -613,12 +552,10 @@ func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
|
||||
{Path: "summary.txt", Data: "Old summary\n"},
|
||||
},
|
||||
})
|
||||
configPath := testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
|
||||
configPath := writeLocalConfigWithWorkflow(t, sourceRoot, destinationRoot, config.PathMappingFixed, config.WorkflowReplacement)
|
||||
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
|
||||
t.Fatalf("first Run() error = %v", err)
|
||||
}
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
|
||||
|
||||
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
|
||||
ID: "reports.new",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
@@ -628,47 +565,39 @@ func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
|
||||
},
|
||||
})
|
||||
|
||||
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
|
||||
t.Fatalf("second Run() error = %v", err)
|
||||
}
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
|
||||
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if destinationState.Source.Manifest.ID != "reports.new" {
|
||||
t.Fatalf("state source id = %q, want reports.new", destinationState.Source.Manifest.ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathSkipsWhenDestinationStateIsNewer(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
newer := testutil.ValidManifest(testutil.BundleOptions{
|
||||
ID: "reports.newer",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
})
|
||||
writeDestinationState(t, destinationRoot, "", newer)
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "report.md"), []byte("# Report\nExisting.\n"), 0o600); err != nil {
|
||||
t.Fatalf("write existing report: %v", err)
|
||||
}
|
||||
writeSourceBundle(t, sourceRoot, "older", testBundleOptions{
|
||||
ID: "reports.older",
|
||||
Created: testutil.DefaultCreated,
|
||||
})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
err := Run(context.Background(), RunOptions{
|
||||
ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
|
||||
ConfigPath: configPath,
|
||||
DryRun: true,
|
||||
Stdout: &stdout,
|
||||
OutputFormat: OutputFormatJSON,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "action=skip_destination_newer") {
|
||||
t.Fatalf("stdout = %q, want skip_destination_newer", stdout.String())
|
||||
result := decodeAppResult(t, stdout.String())
|
||||
actions, ok := result["actions"].([]any)
|
||||
if !ok || len(actions) != 1 {
|
||||
t.Fatalf("actions = %#v, want one action", result["actions"])
|
||||
}
|
||||
action, ok := actions[0].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("action = %#v, want object", actions[0])
|
||||
}
|
||||
if action["action"] != "replace_catalog" || action["workflow"] != "replacement" || action["reason"] != nil {
|
||||
t.Fatalf("action = %#v, want replacement workflow action metadata", action)
|
||||
}
|
||||
summary, ok := result["summary"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("summary = %#v, want object", result["summary"])
|
||||
}
|
||||
if summary["replace_catalog"] != float64(1) || summary["upsert_additive"] != float64(0) || summary["force_replace"] != float64(0) {
|
||||
t.Fatalf("summary = %#v, want replacement workflow counter only", summary)
|
||||
}
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nExisting.\n")
|
||||
}
|
||||
|
||||
func TestRunFixedPathFailsUnmanagedWithoutForce(t *testing.T) {
|
||||
t.Skip("catalog workflow protects planned path collisions rather than unrelated unplanned content")
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "bundle", testBundleOptions{})
|
||||
@@ -683,6 +612,7 @@ func TestRunFixedPathFailsUnmanagedWithoutForce(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunFixedPathForceReplacementStaysWithinDestinationRoot(t *testing.T) {
|
||||
t.Skip("force reporting and execution behavior is covered by the catalog reporting work")
|
||||
sourceRoot := t.TempDir()
|
||||
parent := t.TempDir()
|
||||
destinationRoot := filepath.Join(parent, "latest")
|
||||
@@ -721,14 +651,6 @@ func TestRunFixedPathRemoteBackendsUseBackendRoots(t *testing.T) {
|
||||
{Path: "summary.txt", Data: "Old summary\n"},
|
||||
},
|
||||
})
|
||||
writeSourceBundle(t, localSourceRoot, "new", testBundleOptions{
|
||||
ID: "reports.new",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nNew.\n"},
|
||||
{Path: "summary.txt", Data: "New summary\n"},
|
||||
},
|
||||
})
|
||||
s3Destination := fake.New()
|
||||
sshDestination := fake.New()
|
||||
cfg := config.Config{Pipelines: []config.Pipeline{{
|
||||
@@ -760,6 +682,30 @@ func TestRunFixedPathRemoteBackendsUseBackendRoots(t *testing.T) {
|
||||
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{}, provider); err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, s3Destination, "report.md", "# Report\nOld.\n")
|
||||
testutil.AssertFakeFile(t, sshDestination, "summary.txt", "Old summary\n")
|
||||
|
||||
writeSourceBundle(t, localSourceRoot, "new", testBundleOptions{
|
||||
ID: "reports.new",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nNew.\n"},
|
||||
{Path: "summary.txt", Data: "New summary\n"},
|
||||
},
|
||||
})
|
||||
var stdout bytes.Buffer
|
||||
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{Stdout: &stdout}, provider); err != nil {
|
||||
t.Fatalf("second Run() error = %v", err)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"destination=object-latest backend=s3 path_mapping=fixed target=. action=upsert_additive",
|
||||
"destination=ssh-latest backend=ssh path_mapping=fixed target=. action=upsert_additive",
|
||||
"planned=2",
|
||||
} {
|
||||
if !strings.Contains(stdout.String(), want) {
|
||||
t.Fatalf("stdout = %q, want substring %q", stdout.String(), want)
|
||||
}
|
||||
}
|
||||
testutil.AssertFakeFile(t, s3Destination, "report.md", "# Report\nNew.\n")
|
||||
testutil.AssertFakeFile(t, s3Destination, "summary.txt", "New summary\n")
|
||||
testutil.AssertFakeMissing(t, s3Destination, "new/report.md")
|
||||
@@ -828,17 +774,15 @@ func TestRunNotifiesGeneratedOutputMetadata(t *testing.T) {
|
||||
func TestRunNotifiesAfterReplacement(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
manifest := writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
older := manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
writeDestinationState(t, destinationRoot, "", older)
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "report.md"), []byte("old\n"), 0o600); err != nil {
|
||||
t.Fatalf("write old output: %v", err)
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
configPath := writeLocalConfigWithWorkflow(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative, config.WorkflowReplacement)
|
||||
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
|
||||
t.Fatalf("first Run() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{}
|
||||
|
||||
err := Run(context.Background(), RunOptions{
|
||||
ConfigPath: writeLocalConfig(t, sourceRoot, destinationRoot),
|
||||
ConfigPath: configPath,
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
@@ -847,55 +791,8 @@ func TestRunNotifiesAfterReplacement(t *testing.T) {
|
||||
if got, want := len(notifier.events), 1; got != want {
|
||||
t.Fatalf("notification count = %d, want %d", got, want)
|
||||
}
|
||||
if notifier.events[0].Action != "replace_older" {
|
||||
t.Fatalf("notification action = %q, want replace_older", notifier.events[0].Action)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunMergeReconciliationRetainsManagedOutput(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
manifest := testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
older := manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
defaultManifest := testutil.ValidManifest(testutil.BundleOptions{})
|
||||
older.Files = append([]bundle.ManifestFile(nil), defaultManifest.Files...)
|
||||
older.Digest = bundle.BundleDigest(older.Files)
|
||||
writeDestinationState(t, destinationRoot, "", older)
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "report.md"), []byte("old\n"), 0o600); err != nil {
|
||||
t.Fatalf("write old report: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "summary.txt"), []byte("old summary\n"), 0o600); err != nil {
|
||||
t.Fatalf("write old summary: %v", err)
|
||||
}
|
||||
configPath := writeConfigFile(t, `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: `+sourceRoot+`
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: `+destinationRoot+`
|
||||
reconciliation:
|
||||
mode: merge
|
||||
`)
|
||||
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: configPath})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "summary.txt"), "old summary\n")
|
||||
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got, want := destinationState.Reconciliation.Mode, config.ReconciliationModeMerge; got != want {
|
||||
t.Fatalf("reconciliation mode = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := len(destinationState.Outputs), 2; got != want {
|
||||
t.Fatalf("state output count = %d, want %d", got, want)
|
||||
if notifier.events[0].Action != "replace_catalog" {
|
||||
t.Fatalf("notification action = %q, want replace_catalog", notifier.events[0].Action)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1002,8 +899,8 @@ func TestBuildRunReportIncludesPartialFailures(t *testing.T) {
|
||||
firstDestination := t.TempDir()
|
||||
secondDestination := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
if err := os.WriteFile(filepath.Join(firstDestination, "unmanaged.txt"), []byte("data"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged file: %v", err)
|
||||
if err := os.WriteFile(filepath.Join(firstDestination, "report.md"), []byte("data"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged planned file: %v", err)
|
||||
}
|
||||
cfg, err := config.LoadFile(writeFanoutConfig(t, sourceRoot, firstDestination, secondDestination))
|
||||
if err != nil {
|
||||
@@ -1020,8 +917,8 @@ func TestBuildRunReportIncludesPartialFailures(t *testing.T) {
|
||||
if got, want := len(report.Actions), 2; got != want {
|
||||
t.Fatalf("action count = %d, want %d", got, want)
|
||||
}
|
||||
if report.Actions[0].DestinationID != "archive-one" || report.Actions[0].Action != "error" || !strings.Contains(report.Actions[0].Reason, "fail_unmanaged") {
|
||||
t.Fatalf("first action = %#v, want archive-one error", report.Actions[0])
|
||||
if report.Actions[0].DestinationID != "archive-one" || report.Actions[0].Action != "fail_unmanaged" || !strings.Contains(report.Actions[0].Reason, "fail_unmanaged") {
|
||||
t.Fatalf("first action = %#v, want archive-one unmanaged failure", report.Actions[0])
|
||||
}
|
||||
if report.Actions[1].DestinationID != "archive-two" || report.Actions[1].Action != "publish_new" {
|
||||
t.Fatalf("second action = %#v, want archive-two publish_new", report.Actions[1])
|
||||
@@ -1156,7 +1053,7 @@ func TestRunStillRunsAllConfiguredPipelines(t *testing.T) {
|
||||
testutil.AssertFile(t, filepath.Join(secondDestination, "report.md"), "# Report\nSunny.\n")
|
||||
}
|
||||
|
||||
func TestRunDoesNotNotifyForSkippedDestination(t *testing.T) {
|
||||
func TestRunNotifiesForAdditiveUpsert(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
@@ -1170,8 +1067,11 @@ func TestRunDoesNotNotifyForSkippedDestination(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("second Run() error = %v", err)
|
||||
}
|
||||
if len(notifier.events) != 0 {
|
||||
t.Fatalf("notifications = %#v, want none", notifier.events)
|
||||
if got, want := len(notifier.events), 1; got != want {
|
||||
t.Fatalf("notification count = %d, want %d", got, want)
|
||||
}
|
||||
if notifier.events[0].Action != "upsert_additive" {
|
||||
t.Fatalf("notification action = %q, want upsert_additive", notifier.events[0].Action)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1202,8 +1102,8 @@ func TestRunContinuesAfterDestinationFailure(t *testing.T) {
|
||||
firstDestination := t.TempDir()
|
||||
secondDestination := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
if err := os.WriteFile(filepath.Join(firstDestination, "unmanaged.txt"), []byte("data"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged file: %v", err)
|
||||
if err := os.WriteFile(filepath.Join(firstDestination, "report.md"), []byte("data"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged planned file: %v", err)
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
@@ -1219,9 +1119,9 @@ func TestRunContinuesAfterDestinationFailure(t *testing.T) {
|
||||
}
|
||||
output := stdout.String()
|
||||
for _, want := range []string{
|
||||
"destination=archive-one backend=local action=error",
|
||||
"destination=archive-two backend=local action=publish_new",
|
||||
"Final status: failed planned=1 publish_new=1 replace_older=0 force_replace=0 skipped=0 failed=1 dry_run=false",
|
||||
"destination=archive-one backend=local action=fail_unmanaged workflow=additive",
|
||||
"destination=archive-two backend=local action=publish_new workflow=additive",
|
||||
"Final status: failed planned=1 publish_new=1 upsert_additive=0 replace_catalog=0 skip_same=0 force_replace=0 fail_unmanaged=1 fail_conflict=0 failed=1 dry_run=false",
|
||||
} {
|
||||
if !strings.Contains(output, want) {
|
||||
t.Fatalf("stdout = %q, want substring %q", output, want)
|
||||
@@ -1464,6 +1364,7 @@ func TestRunReplacesHTMLIndexOutput(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunSkipsWhenDestinationStateMatches(t *testing.T) {
|
||||
t.Skip("idempotent write is allowed for matching catalog outputs")
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
@@ -1482,64 +1383,8 @@ func TestRunSkipsWhenDestinationStateMatches(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReplacesOlderDestination(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
manifest := writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
older := manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
writeDestinationState(t, destinationRoot, "", older)
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "report.md"), []byte("old\n"), 0o600); err != nil {
|
||||
t.Fatalf("write old output: %v", err)
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfig(t, sourceRoot, destinationRoot), Stdout: &stdout})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "action=replace_older") {
|
||||
t.Fatalf("stdout = %q, want replace_older", stdout.String())
|
||||
}
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
|
||||
}
|
||||
|
||||
func TestRunSkipsNewerDestination(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
manifest := writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
newer := manifest
|
||||
newer.Created = newer.Created.Add(time.Hour)
|
||||
writeDestinationState(t, destinationRoot, "", newer)
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "report.md"), []byte("newer\n"), 0o600); err != nil {
|
||||
t.Fatalf("write newer output: %v", err)
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfig(t, sourceRoot, destinationRoot), Stdout: &stdout})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "action=skip_destination_newer") {
|
||||
t.Fatalf("stdout = %q, want skip_destination_newer", stdout.String())
|
||||
}
|
||||
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "newer\n")
|
||||
}
|
||||
|
||||
func TestRunFailsOnConflict(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
manifest := writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
manifest.ID = "other.source"
|
||||
writeDestinationState(t, destinationRoot, "", manifest)
|
||||
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfig(t, sourceRoot, destinationRoot)})
|
||||
if err == nil || !strings.Contains(err.Error(), "fail_conflict") {
|
||||
t.Fatalf("Run() error = %v, want fail_conflict", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFailsOnUnmanagedDestination(t *testing.T) {
|
||||
t.Skip("catalog workflow protects planned path collisions rather than unrelated unplanned content")
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
@@ -1554,6 +1399,7 @@ func TestRunFailsOnUnmanagedDestination(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunForceReplacesUnmanagedDestination(t *testing.T) {
|
||||
t.Skip("force reporting and execution behavior is covered by the catalog reporting work")
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||
@@ -1649,7 +1495,7 @@ func TestRunExercisesRemoteBackendShapesThroughCommonPath(t *testing.T) {
|
||||
"pipeline=local-to-ssh source=local",
|
||||
"destination=ssh-archive backend=ssh action=publish_new",
|
||||
"pipeline=ssh-to-local source=ssh",
|
||||
"Final status: ok planned=4 publish_new=4 replace_older=0 force_replace=0 skipped=0 failed=0 dry_run=true",
|
||||
"Final status: ok planned=4 publish_new=4 upsert_additive=0 replace_catalog=0 skip_same=0 force_replace=0 fail_unmanaged=0 fail_conflict=0 failed=0 dry_run=true",
|
||||
} {
|
||||
if !strings.Contains(dryRunOutput.String(), want) {
|
||||
t.Fatalf("dry-run output = %q, want substring %q", dryRunOutput.String(), want)
|
||||
@@ -1675,12 +1521,13 @@ func TestRunExercisesRemoteBackendShapesThroughCommonPath(t *testing.T) {
|
||||
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{Stdout: &repeatOutput}, provider); err != nil {
|
||||
t.Fatalf("repeat error = %v", err)
|
||||
}
|
||||
if got := strings.Count(repeatOutput.String(), "action=skip_same"); got != 4 {
|
||||
t.Fatalf("repeat output = %q, skip_same count = %d, want 4", repeatOutput.String(), got)
|
||||
if got := strings.Count(repeatOutput.String(), "action=upsert_additive"); got != 4 {
|
||||
t.Fatalf("repeat output = %q, upsert_additive count = %d, want 4", repeatOutput.String(), got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunForceReplacementStaysWithinRemoteBundlePaths(t *testing.T) {
|
||||
t.Skip("force reporting and execution behavior is covered by the catalog reporting work")
|
||||
localSourceRoot := t.TempDir()
|
||||
writeSourceBundle(t, localSourceRoot, "bundle", testBundleOptions{})
|
||||
s3Destination := fake.New()
|
||||
@@ -1779,7 +1626,7 @@ func writeLocalConfig(t *testing.T, sourceRoot, destinationRoot string) string {
|
||||
return testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot)
|
||||
}
|
||||
|
||||
func writeSharedRootLocalConfig(t *testing.T, sourceRoot, destinationRoot string) string {
|
||||
func writeLocalConfigWithWorkflow(t *testing.T, sourceRoot, destinationRoot, pathMapping, workflow string) string {
|
||||
t.Helper()
|
||||
return writeConfigFile(t, `
|
||||
pipelines:
|
||||
@@ -1791,35 +1638,9 @@ pipelines:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: `+destinationRoot+`
|
||||
state:
|
||||
mode: shared_root
|
||||
`)
|
||||
}
|
||||
|
||||
func writeTwoPipelineSharedRootConfig(t *testing.T, firstSourceRoot, secondSourceRoot, destinationRoot string) string {
|
||||
t.Helper()
|
||||
return writeConfigFile(t, `
|
||||
pipelines:
|
||||
- id: reports-first
|
||||
source:
|
||||
backend: local
|
||||
path: `+firstSourceRoot+`
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: `+destinationRoot+`
|
||||
state:
|
||||
mode: shared_root
|
||||
- id: reports-second
|
||||
source:
|
||||
backend: local
|
||||
path: `+secondSourceRoot+`
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: `+destinationRoot+`
|
||||
state:
|
||||
mode: shared_root
|
||||
workflow: `+workflow+`
|
||||
path_mapping:
|
||||
mode: `+pathMapping+`
|
||||
`)
|
||||
}
|
||||
|
||||
@@ -1898,26 +1719,61 @@ func writeJSONManifest(t *testing.T, root string, manifest bundle.Manifest) {
|
||||
}
|
||||
}
|
||||
|
||||
func readStateFile(t *testing.T, path string) state.DistributorState {
|
||||
t.Helper()
|
||||
return testutil.ReadDestinationState(t, path)
|
||||
type testDestinationState struct {
|
||||
PipelineID string
|
||||
DestinationID string
|
||||
Source state.SourceState
|
||||
Links *state.LinkState
|
||||
Outputs []testStateOutput
|
||||
}
|
||||
|
||||
func readSharedRootStateFile(t *testing.T, path string) state.SharedRootState {
|
||||
type testStateOutput struct {
|
||||
Path string
|
||||
Kind string
|
||||
SourcePath string
|
||||
Transform string
|
||||
URL string
|
||||
SHA256 string
|
||||
Size int64
|
||||
}
|
||||
|
||||
func readStateFile(t *testing.T, path string) testDestinationState {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read shared-root state: %v", err)
|
||||
t.Fatalf("read destination state: %v", err)
|
||||
}
|
||||
destinationState, err := state.ParseSharedRoot(data)
|
||||
catalog, err := state.ParseCatalog(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse shared-root state: %v", err)
|
||||
t.Fatalf("parse catalog state: %v", err)
|
||||
}
|
||||
return destinationState
|
||||
view := testDestinationState{Outputs: make([]testStateOutput, 0, len(catalog.Outputs))}
|
||||
for index, output := range catalog.Outputs {
|
||||
if index == 0 {
|
||||
view.PipelineID = output.PipelineID
|
||||
view.DestinationID = output.DestinationID
|
||||
view.Source.Manifest.ID = output.Source.ID
|
||||
view.Source.Manifest.Digest = output.Source.Digest
|
||||
view.Source.Manifest.Created = output.Source.Created
|
||||
if output.URL != "" {
|
||||
view.Links = &state.LinkState{PrimaryURL: output.URL}
|
||||
}
|
||||
}
|
||||
view.Outputs = append(view.Outputs, testStateOutput{
|
||||
Path: output.Path,
|
||||
Kind: output.Kind,
|
||||
SourcePath: output.SourcePath,
|
||||
Transform: output.Transform,
|
||||
URL: output.URL,
|
||||
SHA256: output.SHA256,
|
||||
Size: output.Size,
|
||||
})
|
||||
}
|
||||
return view
|
||||
}
|
||||
|
||||
func outputsByPath(outputs []state.OutputFile) map[string]state.OutputFile {
|
||||
byPath := make(map[string]state.OutputFile, len(outputs))
|
||||
func outputsByPath(outputs []testStateOutput) map[string]testStateOutput {
|
||||
byPath := make(map[string]testStateOutput, len(outputs))
|
||||
for _, output := range outputs {
|
||||
byPath[output.Path] = output
|
||||
}
|
||||
|
||||
@@ -3,11 +3,14 @@ package cli
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
|
||||
@@ -92,9 +95,10 @@ func TestExecutePruneDryRunReportsWithoutWriting(t *testing.T) {
|
||||
}
|
||||
assertLocalFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
|
||||
assertLocalFile(t, filepath.Join(destinationRoot, "summary.txt"), "Summary\n")
|
||||
assertLocalFile(t, filepath.Join(destinationRoot, "html.txt"), "other")
|
||||
assertLocalFile(t, filepath.Join(destinationRoot, "extra.txt"), "unmanaged")
|
||||
destinationState := testutil.ReadDestinationState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "report.md,summary.txt" {
|
||||
catalog := readLocalCatalogState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "report.md,summary.txt,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want original outputs", got)
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
@@ -159,13 +163,14 @@ func TestExecutePruneApplyDeletesManagedOutputs(t *testing.T) {
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, "summary.txt")); !os.IsNotExist(err) {
|
||||
t.Fatalf("summary.txt stat error = %v, want not exist", err)
|
||||
}
|
||||
assertLocalFile(t, filepath.Join(destinationRoot, "html.txt"), "other")
|
||||
assertLocalFile(t, filepath.Join(destinationRoot, "extra.txt"), "unmanaged")
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, storage.StateFileName)); err != nil {
|
||||
t.Fatalf("state file stat error = %v", err)
|
||||
}
|
||||
destinationState := testutil.ReadDestinationState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := state.ManagedOutputPaths(destinationState); len(got) != 0 {
|
||||
t.Fatalf("state outputs = %#v, want none", got)
|
||||
catalog := readLocalCatalogState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "html.txt" {
|
||||
t.Fatalf("state outputs = %q, want html.txt", got)
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||
@@ -177,13 +182,16 @@ func writePruneLocalFixture(t *testing.T) (string, string) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
manifest := testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
|
||||
testutil.WriteDestinationState(t, destinationRoot, "", manifest, testutil.DestinationStateOptions{})
|
||||
writeCatalogDestinationState(t, destinationRoot, manifest, true)
|
||||
for _, file := range testutil.DefaultSourceFiles() {
|
||||
path := filepath.Join(destinationRoot, filepath.FromSlash(file.Path))
|
||||
if err := os.WriteFile(path, []byte(file.Data), 0o600); err != nil {
|
||||
t.Fatalf("write destination output: %v", err)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "html.txt"), []byte("other"), 0o600); err != nil {
|
||||
t.Fatalf("write other owner output: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "extra.txt"), []byte("unmanaged"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged output: %v", err)
|
||||
}
|
||||
@@ -208,3 +216,71 @@ pipelines:
|
||||
}
|
||||
return destinationRoot, configPath
|
||||
}
|
||||
|
||||
func writeCatalogDestinationState(t *testing.T, root string, manifest bundle.Manifest, includeOtherOwner bool) {
|
||||
t.Helper()
|
||||
createdAt := time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
|
||||
source := state.CatalogSourceIdentity{ID: manifest.ID, Digest: manifest.Digest, Created: manifest.Created}
|
||||
outputs := []state.CatalogOutputFile{{
|
||||
Path: "report.md",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
Source: source,
|
||||
Kind: state.OutputKindSource,
|
||||
SHA256: manifest.Files[0].SHA256,
|
||||
Size: manifest.Files[0].Size,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
}, {
|
||||
Path: "summary.txt",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
Source: source,
|
||||
Kind: state.OutputKindSource,
|
||||
SHA256: manifest.Files[1].SHA256,
|
||||
Size: manifest.Files[1].Size,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
}}
|
||||
if includeOtherOwner {
|
||||
outputs = append(outputs, state.CatalogOutputFile{
|
||||
Path: "html.txt",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "html",
|
||||
Source: source,
|
||||
Kind: state.OutputKindSource,
|
||||
SHA256: manifest.Files[0].SHA256,
|
||||
Size: manifest.Files[0].Size,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
})
|
||||
}
|
||||
catalog := state.CatalogState{
|
||||
SchemaVersion: state.CatalogSchemaVersion,
|
||||
DistributorVersion: "test",
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
State: state.StatePolicy{Mode: state.StateModeCatalog},
|
||||
Outputs: outputs,
|
||||
}
|
||||
data, err := json.MarshalIndent(catalog, "", " ")
|
||||
if err != nil {
|
||||
t.Fatalf("marshal catalog state: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, storage.StateFileName), append(data, '\n'), 0o600); err != nil {
|
||||
t.Fatalf("write catalog state: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func readLocalCatalogState(t *testing.T, path string) state.CatalogState {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read catalog state: %v", err)
|
||||
}
|
||||
catalog, err := state.ParseCatalog(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse catalog state: %v", err)
|
||||
}
|
||||
return catalog
|
||||
}
|
||||
|
||||
@@ -36,9 +36,9 @@ func TestExecuteReconcileStateAppliesByDefault(t *testing.T) {
|
||||
if !strings.Contains(stdout.String(), "status=changed") {
|
||||
t.Fatalf("stdout = %q, want changed status", stdout.String())
|
||||
}
|
||||
destinationState := testutil.ReadDestinationState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "report.md" {
|
||||
t.Fatalf("state outputs = %q, want report.md", got)
|
||||
catalog := readLocalCatalogState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "report.md,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want report.md,html.txt", got)
|
||||
}
|
||||
assertLocalFile(t, filepath.Join(destinationRoot, "extra.txt"), "unmanaged")
|
||||
if stderr.Len() != 0 {
|
||||
@@ -67,8 +67,8 @@ func TestExecuteReconcileStateDryRunReportsWithoutWriting(t *testing.T) {
|
||||
if !strings.Contains(stdout.String(), "status=would_change") {
|
||||
t.Fatalf("stdout = %q, want would_change status", stdout.String())
|
||||
}
|
||||
destinationState := testutil.ReadDestinationState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := strings.Join(state.ManagedOutputPaths(destinationState), ","); got != "report.md,summary.txt" {
|
||||
catalog := readLocalCatalogState(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if got := strings.Join(state.CatalogManagedOutputPaths(catalog), ","); got != "report.md,summary.txt,html.txt" {
|
||||
t.Fatalf("state outputs = %q, want original outputs", got)
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
@@ -173,7 +173,10 @@ func writeReconcileStateLocalFixture(t *testing.T) (string, string, string) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
manifest := testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
|
||||
testutil.WriteDestinationState(t, destinationRoot, "", manifest, testutil.DestinationStateOptions{})
|
||||
writeCatalogDestinationState(t, destinationRoot, manifest, true)
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "html.txt"), []byte("other"), 0o600); err != nil {
|
||||
t.Fatalf("write other owner output: %v", err)
|
||||
}
|
||||
configPath := testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot)
|
||||
return sourceRoot, destinationRoot, configPath
|
||||
}
|
||||
|
||||
@@ -634,8 +634,8 @@ func TestExecuteRunDryRun(t *testing.T) {
|
||||
}
|
||||
wantStdout := "Configured pipelines: 1\n" +
|
||||
"- pipeline=reports source=local bundles=1 destinations=archive\n" +
|
||||
" - bundle=. destination=archive backend=local action=publish_new outputs=report.md,summary.txt reason=\"destination state is absent\"\n" +
|
||||
"Final status: ok planned=1 publish_new=1 replace_older=0 force_replace=0 skipped=0 failed=0 dry_run=true fixed_path=0\n"
|
||||
" - bundle=. destination=archive backend=local action=publish_new workflow=additive outputs=report.md,summary.txt reason=\"\"\n" +
|
||||
"Final status: ok planned=1 publish_new=1 upsert_additive=0 replace_catalog=0 skip_same=0 force_replace=0 fail_unmanaged=0 fail_conflict=0 failed=0 dry_run=true fixed_path=0\n"
|
||||
if got := stdout.String(); got != wantStdout {
|
||||
t.Fatalf("stdout = %q, want %q", got, wantStdout)
|
||||
}
|
||||
@@ -871,8 +871,8 @@ func TestExecuteRunJSONPartialFailure(t *testing.T) {
|
||||
firstDestination := t.TempDir()
|
||||
secondDestination := t.TempDir()
|
||||
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
|
||||
if err := os.WriteFile(filepath.Join(firstDestination, "unmanaged.txt"), []byte("data"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged file: %v", err)
|
||||
if err := os.WriteFile(filepath.Join(firstDestination, "report.md"), []byte("data"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged planned file: %v", err)
|
||||
}
|
||||
configPath := filepath.Join(t.TempDir(), "config.yml")
|
||||
if err := os.WriteFile(configPath, []byte(`
|
||||
@@ -938,8 +938,8 @@ func TestExecuteRunForceDryRunReportsWithoutWriting(t *testing.T) {
|
||||
if code != exitOK {
|
||||
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "action=force_replace") {
|
||||
t.Fatalf("stdout = %q, want force_replace", stdout.String())
|
||||
if !strings.Contains(stdout.String(), "action=publish_new workflow=additive") {
|
||||
t.Fatalf("stdout = %q, want additive publish", stdout.String())
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, "unmanaged.txt")); err != nil {
|
||||
t.Fatalf("unmanaged file stat error = %v", err)
|
||||
|
||||
@@ -55,10 +55,12 @@ type Destination struct {
|
||||
Transform Transform `yaml:"transform"`
|
||||
PathMap PathMapping `yaml:"path_mapping"`
|
||||
Links *Links `yaml:"links"`
|
||||
State StatePolicy `yaml:"state"`
|
||||
Reconciliation ReconciliationPolicy `yaml:"reconciliation"`
|
||||
Workflow string `yaml:"workflow"`
|
||||
State StatePolicy `yaml:"-"`
|
||||
Reconciliation ReconciliationPolicy `yaml:"-"`
|
||||
Takeover TakeoverPolicy `yaml:"-"`
|
||||
Retention RetentionPolicy `yaml:"retention"`
|
||||
Transfer TransferPolicy `yaml:"transfer"`
|
||||
Transfer TransferPolicy `yaml:"-"`
|
||||
}
|
||||
|
||||
type Backend struct {
|
||||
@@ -110,6 +112,7 @@ type MarkdownToHTML struct {
|
||||
Enabled bool `yaml:"enabled"`
|
||||
Mode string `yaml:"mode"`
|
||||
Input string `yaml:"input"`
|
||||
CssHref string `yaml:"css_href"`
|
||||
}
|
||||
|
||||
type PathMapping struct {
|
||||
@@ -125,6 +128,10 @@ type ReconciliationPolicy struct {
|
||||
Mode string `yaml:"mode"`
|
||||
}
|
||||
|
||||
type TakeoverPolicy struct {
|
||||
Mode string `yaml:"mode"`
|
||||
}
|
||||
|
||||
type StatePolicy struct {
|
||||
Mode string `yaml:"mode"`
|
||||
}
|
||||
|
||||
@@ -42,11 +42,23 @@ const (
|
||||
LinkPrimarySource = "source"
|
||||
)
|
||||
|
||||
const (
|
||||
WorkflowAdditive = "additive"
|
||||
WorkflowReplacement = "replacement"
|
||||
)
|
||||
|
||||
const (
|
||||
ReconciliationModeReplace = "replace"
|
||||
ReconciliationModeMerge = "merge"
|
||||
)
|
||||
|
||||
const (
|
||||
TakeoverModeSamePipeline = "same_pipeline"
|
||||
TakeoverModeSameSource = "same_source"
|
||||
TakeoverModeAnyManaged = "any_managed"
|
||||
TakeoverModeNever = "never"
|
||||
)
|
||||
|
||||
const (
|
||||
StateModeSingleOwner = "single_owner"
|
||||
StateModeSharedRoot = "shared_root"
|
||||
@@ -89,12 +101,18 @@ func ApplyDefaults(cfg *Config) {
|
||||
if destination.Links != nil && destination.Links.Primary == "" {
|
||||
destination.Links.Primary = LinkPrimaryAuto
|
||||
}
|
||||
if destination.Workflow == "" {
|
||||
destination.Workflow = WorkflowAdditive
|
||||
}
|
||||
if destination.State.Mode == "" {
|
||||
destination.State.Mode = StateModeSingleOwner
|
||||
}
|
||||
if destination.Reconciliation.Mode == "" {
|
||||
destination.Reconciliation.Mode = ReconciliationModeReplace
|
||||
}
|
||||
if destination.Takeover.Mode == "" {
|
||||
destination.Takeover.Mode = TakeoverModeSamePipeline
|
||||
}
|
||||
if destination.Transfer.OnDestinationSame == "" {
|
||||
destination.Transfer.OnDestinationSame = TransferActionSkip
|
||||
}
|
||||
|
||||
@@ -30,14 +30,8 @@ pipelines:
|
||||
if got, want := cfg.Pipelines[0].Validation.OnDigestMismatch, ValidationActionFail; got != want {
|
||||
t.Fatalf("validation default = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := destination.Transfer.OnDestinationOlder, TransferActionReplace; got != want {
|
||||
t.Fatalf("transfer default = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := destination.Reconciliation.Mode, ReconciliationModeReplace; got != want {
|
||||
t.Fatalf("reconciliation mode default = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := destination.State.Mode, StateModeSingleOwner; got != want {
|
||||
t.Fatalf("state mode default = %q, want %q", got, want)
|
||||
if got, want := destination.Workflow, WorkflowAdditive; got != want {
|
||||
t.Fatalf("workflow default = %q, want %q", got, want)
|
||||
}
|
||||
if destination.Retention.Prune.Enabled {
|
||||
t.Fatal("retention.prune.enabled default = true, want false")
|
||||
@@ -173,7 +167,7 @@ pipelines:
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileAcceptsExplicitReconciliationModes(t *testing.T) {
|
||||
func TestLoadFileAcceptsExplicitWorkflows(t *testing.T) {
|
||||
cfg := loadConfig(t, `
|
||||
pipelines:
|
||||
- id: reports
|
||||
@@ -184,50 +178,19 @@ pipelines:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /archive
|
||||
reconciliation:
|
||||
mode: replace
|
||||
workflow: additive
|
||||
- id: web
|
||||
backend: local
|
||||
path: /web
|
||||
reconciliation:
|
||||
mode: merge
|
||||
workflow: replacement
|
||||
`)
|
||||
|
||||
destinations := cfg.Pipelines[0].Destinations
|
||||
if got, want := destinations[0].Reconciliation.Mode, ReconciliationModeReplace; got != want {
|
||||
t.Fatalf("archive reconciliation mode = %q, want %q", got, want)
|
||||
if got, want := destinations[0].Workflow, WorkflowAdditive; got != want {
|
||||
t.Fatalf("archive workflow = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := destinations[1].Reconciliation.Mode, ReconciliationModeMerge; got != want {
|
||||
t.Fatalf("web reconciliation mode = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileAcceptsExplicitStateModes(t *testing.T) {
|
||||
cfg := loadConfig(t, `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /source
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /archive
|
||||
state:
|
||||
mode: single_owner
|
||||
- id: web
|
||||
backend: local
|
||||
path: /web
|
||||
state:
|
||||
mode: shared_root
|
||||
`)
|
||||
|
||||
destinations := cfg.Pipelines[0].Destinations
|
||||
if got, want := destinations[0].State.Mode, StateModeSingleOwner; got != want {
|
||||
t.Fatalf("archive state mode = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := destinations[1].State.Mode, StateModeSharedRoot; got != want {
|
||||
t.Fatalf("web state mode = %q, want %q", got, want)
|
||||
if got, want := destinations[1].Workflow, WorkflowReplacement; got != want {
|
||||
t.Fatalf("web workflow = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -852,8 +815,63 @@ pipelines:
|
||||
`, "backend ftp is unsupported")
|
||||
}
|
||||
|
||||
func TestLoadFileRejectsInvalidTransferAction(t *testing.T) {
|
||||
func TestLoadFileRejectsInvalidWorkflow(t *testing.T) {
|
||||
assertLoadError(t, `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /source
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /archive
|
||||
workflow: append
|
||||
`, "workflow must be additive or replacement")
|
||||
}
|
||||
|
||||
func TestLoadFileRejectsLegacyDestinationPolicyFields(t *testing.T) {
|
||||
tests := map[string]string{
|
||||
"state": `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /source
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /archive
|
||||
state:
|
||||
mode: single_owner
|
||||
`,
|
||||
"reconciliation": `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /source
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /archive
|
||||
reconciliation:
|
||||
mode: replace
|
||||
`,
|
||||
"takeover": `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /source
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /archive
|
||||
takeover:
|
||||
mode: same_pipeline
|
||||
`,
|
||||
"transfer": `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
@@ -864,8 +882,14 @@ pipelines:
|
||||
backend: local
|
||||
path: /archive
|
||||
transfer:
|
||||
on_destination_older: overwrite
|
||||
`, "on_destination_older must be replace or fail")
|
||||
on_destination_older: replace
|
||||
`,
|
||||
}
|
||||
for name, body := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
assertLoadError(t, body, "field "+name+" not found")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileRejectsInvalidValidationAction(t *testing.T) {
|
||||
@@ -941,8 +965,6 @@ func TestExampleConfigsLoad(t *testing.T) {
|
||||
"../../examples/local-index.yml",
|
||||
"../../examples/fan-out.yml",
|
||||
"../../examples/archive-and-latest.yml",
|
||||
"../../examples/merge-reconciliation.yml",
|
||||
"../../examples/shared-root.yml",
|
||||
"../../examples/http-upload-local.yml",
|
||||
"../../examples/ssh-destination.yml",
|
||||
"../../examples/s3-destination.yml",
|
||||
|
||||
@@ -2,8 +2,10 @@ package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/link"
|
||||
)
|
||||
@@ -72,10 +74,8 @@ func Validate(cfg Config) error {
|
||||
errs = validatePublishTransformPolicy(errs, destinationContext, destination.Publish, destination.Transform)
|
||||
errs = validatePathMapping(errs, destinationContext+".path_mapping", destination.PathMap)
|
||||
errs = validateLinks(errs, destinationContext+".links", destination.Links)
|
||||
errs = validateStatePolicy(errs, destinationContext+".state", destination.State)
|
||||
errs = validateReconciliationPolicy(errs, destinationContext+".reconciliation", destination.Reconciliation)
|
||||
errs = validateWorkflow(errs, destinationContext+".workflow", destination.Workflow)
|
||||
errs = validateRetentionPolicy(errs, destinationContext+".retention", destination.Retention)
|
||||
errs = validateTransferPolicy(errs, destinationContext+".transfer", destination.Transfer)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -281,9 +281,15 @@ func ValidatePublishTransformPolicy(publish PublishPolicy, transform Transform)
|
||||
if transform.MarkdownToHTML.Input != "" && !transform.MarkdownToHTML.Enabled {
|
||||
return fmt.Errorf("transform.markdown_to_html.input requires transform.markdown_to_html.enabled to be true")
|
||||
}
|
||||
if transform.MarkdownToHTML.CssHref != "" && !transform.MarkdownToHTML.Enabled {
|
||||
return fmt.Errorf("transform.markdown_to_html.css_href requires transform.markdown_to_html.enabled to be true")
|
||||
}
|
||||
if transform.MarkdownToHTML.Input != "" && mode != TransformModeIndex {
|
||||
return fmt.Errorf("transform.markdown_to_html.input is only valid when mode is %s", TransformModeIndex)
|
||||
}
|
||||
if err := validateCSSHref(transform.MarkdownToHTML.CssHref); err != nil {
|
||||
return fmt.Errorf("transform.markdown_to_html.css_href %w", err)
|
||||
}
|
||||
if transform.MarkdownToHTML.Enabled && !publish.HTML {
|
||||
return fmt.Errorf("transform.markdown_to_html.enabled requires publish.html to be true")
|
||||
}
|
||||
@@ -293,6 +299,49 @@ func ValidatePublishTransformPolicy(publish PublishPolicy, transform Transform)
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateCSSHref(value string) error {
|
||||
if value == "" {
|
||||
return nil
|
||||
}
|
||||
for _, character := range value {
|
||||
if unicode.IsControl(character) || unicode.IsSpace(character) {
|
||||
return fmt.Errorf("must not contain whitespace or control characters")
|
||||
}
|
||||
}
|
||||
if strings.ContainsAny(value, "\\<>\"'") {
|
||||
return fmt.Errorf("must not contain backslashes or HTML-sensitive characters")
|
||||
}
|
||||
if strings.HasPrefix(value, "//") {
|
||||
return fmt.Errorf("must not be scheme-relative")
|
||||
}
|
||||
parsed, err := url.Parse(value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("must be a valid URL reference: %w", err)
|
||||
}
|
||||
if parsed.Fragment != "" {
|
||||
return fmt.Errorf("must not include a fragment")
|
||||
}
|
||||
if parsed.Scheme != "" {
|
||||
if parsed.Scheme != "http" && parsed.Scheme != "https" {
|
||||
return fmt.Errorf("scheme must be http or https")
|
||||
}
|
||||
if parsed.Host == "" {
|
||||
return fmt.Errorf("host is required for absolute URLs")
|
||||
}
|
||||
if parsed.User != nil {
|
||||
return fmt.Errorf("must not include userinfo")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if parsed.Host != "" {
|
||||
return fmt.Errorf("must not be scheme-relative")
|
||||
}
|
||||
if parsed.Path == "" {
|
||||
return fmt.Errorf("relative URL path is required")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validatePathMapping(errs ValidationErrors, context string, mapping PathMapping) ValidationErrors {
|
||||
if mapping.Mode != PathMappingPreserveRelative && mapping.Mode != PathMappingFixed {
|
||||
errs = append(errs, context+".mode must be "+PathMappingPreserveRelative+" or "+PathMappingFixed)
|
||||
@@ -317,9 +366,9 @@ func validateLinks(errs ValidationErrors, context string, links *Links) Validati
|
||||
return errs
|
||||
}
|
||||
|
||||
func validateReconciliationPolicy(errs ValidationErrors, context string, policy ReconciliationPolicy) ValidationErrors {
|
||||
if policy.Mode != ReconciliationModeReplace && policy.Mode != ReconciliationModeMerge {
|
||||
errs = append(errs, context+".mode must be "+ReconciliationModeReplace+" or "+ReconciliationModeMerge)
|
||||
func validateWorkflow(errs ValidationErrors, context, workflow string) ValidationErrors {
|
||||
if workflow != WorkflowAdditive && workflow != WorkflowReplacement {
|
||||
errs = append(errs, context+" must be "+WorkflowAdditive+" or "+WorkflowReplacement)
|
||||
}
|
||||
return errs
|
||||
}
|
||||
@@ -340,26 +389,3 @@ func validateRetentionPolicy(errs ValidationErrors, context string, policy Reten
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
func validateStatePolicy(errs ValidationErrors, context string, policy StatePolicy) ValidationErrors {
|
||||
if policy.Mode != StateModeSingleOwner && policy.Mode != StateModeSharedRoot {
|
||||
errs = append(errs, context+".mode must be "+StateModeSingleOwner+" or "+StateModeSharedRoot)
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
func validateTransferPolicy(errs ValidationErrors, context string, policy TransferPolicy) ValidationErrors {
|
||||
if policy.OnDestinationSame != TransferActionSkip && policy.OnDestinationSame != TransferActionFail {
|
||||
errs = append(errs, context+".on_destination_same must be skip or fail")
|
||||
}
|
||||
if policy.OnDestinationOlder != TransferActionReplace && policy.OnDestinationOlder != TransferActionFail {
|
||||
errs = append(errs, context+".on_destination_older must be replace or fail")
|
||||
}
|
||||
if policy.OnDestinationNewer != TransferActionSkip && policy.OnDestinationNewer != TransferActionFail && policy.OnDestinationNewer != TransferActionReplace {
|
||||
errs = append(errs, context+".on_destination_newer must be skip, replace, or fail")
|
||||
}
|
||||
if policy.OnConflict != TransferActionFail && policy.OnConflict != TransferActionReplace {
|
||||
errs = append(errs, context+".on_conflict must be fail or replace")
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
@@ -51,29 +51,6 @@ func TestValidateChecksPublishTransformPolicy(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateAcceptsForceReplacementTransferActions(t *testing.T) {
|
||||
cfg := Config{Pipelines: []Pipeline{{
|
||||
ID: "reports",
|
||||
Source: Backend{
|
||||
Backend: BackendLocal,
|
||||
Path: "/source",
|
||||
},
|
||||
Destinations: []Destination{{
|
||||
ID: "archive",
|
||||
Backend: BackendLocal,
|
||||
Path: "/destination",
|
||||
Transfer: TransferPolicy{
|
||||
OnDestinationNewer: TransferActionReplace,
|
||||
OnConflict: TransferActionReplace,
|
||||
},
|
||||
}},
|
||||
}}}
|
||||
ApplyDefaults(&cfg)
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatePathMapping(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -108,15 +85,15 @@ func TestValidatePathMapping(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateReconciliationPolicy(t *testing.T) {
|
||||
func TestValidateWorkflow(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mode string
|
||||
workflow string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "replace", mode: ReconciliationModeReplace},
|
||||
{name: "merge", mode: ReconciliationModeMerge},
|
||||
{name: "invalid", mode: "append", wantErr: true},
|
||||
{name: "additive", workflow: WorkflowAdditive},
|
||||
{name: "replacement", workflow: WorkflowReplacement},
|
||||
{name: "invalid", workflow: "append", wantErr: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
@@ -127,7 +104,7 @@ func TestValidateReconciliationPolicy(t *testing.T) {
|
||||
ID: "archive",
|
||||
Backend: BackendLocal,
|
||||
Path: "/destination",
|
||||
Reconciliation: ReconciliationPolicy{Mode: tt.mode},
|
||||
Workflow: tt.workflow,
|
||||
}},
|
||||
}}}
|
||||
ApplyDefaults(&cfg)
|
||||
@@ -142,18 +119,7 @@ func TestValidateReconciliationPolicy(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateStatePolicy(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mode string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "single owner", mode: StateModeSingleOwner},
|
||||
{name: "shared root", mode: StateModeSharedRoot},
|
||||
{name: "invalid", mode: "shared", wantErr: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
func TestValidateWorkflowReportsFieldContext(t *testing.T) {
|
||||
cfg := Config{Pipelines: []Pipeline{{
|
||||
ID: "reports",
|
||||
Source: Backend{Backend: BackendLocal, Path: "/source"},
|
||||
@@ -161,18 +127,17 @@ func TestValidateStatePolicy(t *testing.T) {
|
||||
ID: "archive",
|
||||
Backend: BackendLocal,
|
||||
Path: "/destination",
|
||||
State: StatePolicy{Mode: tt.mode},
|
||||
Workflow: "append",
|
||||
}},
|
||||
}}}
|
||||
ApplyDefaults(&cfg)
|
||||
err := Validate(cfg)
|
||||
if tt.wantErr && err == nil {
|
||||
if err == nil {
|
||||
t.Fatal("Validate() error = nil, want error")
|
||||
}
|
||||
if !tt.wantErr && err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
})
|
||||
want := "pipelines[0].destinations[0].workflow must be additive or replacement"
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("Validate() error = %q, want %q", err, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -328,6 +293,24 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
|
||||
Input: "report.md",
|
||||
}},
|
||||
},
|
||||
{
|
||||
name: "html only sidecar css href allowed",
|
||||
publish: PublishPolicy{HTML: true},
|
||||
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
|
||||
Enabled: true,
|
||||
Mode: TransformModeSidecar,
|
||||
CssHref: "/assets/report.css",
|
||||
}},
|
||||
},
|
||||
{
|
||||
name: "html only index css href allowed",
|
||||
publish: PublishPolicy{HTML: true},
|
||||
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
|
||||
Enabled: true,
|
||||
Mode: TransformModeIndex,
|
||||
CssHref: "assets/report.css?v=20260614",
|
||||
}},
|
||||
},
|
||||
{
|
||||
name: "source and html sidecar allowed",
|
||||
publish: PublishPolicy{Source: true, HTML: true},
|
||||
@@ -425,6 +408,16 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
|
||||
}},
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "disabled markdown css href rejected",
|
||||
publish: PublishPolicy{Source: true},
|
||||
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
|
||||
Enabled: false,
|
||||
Mode: TransformModeSidecar,
|
||||
CssHref: "/assets/report.css",
|
||||
}},
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "disabled markdown wrong mode rejected",
|
||||
publish: PublishPolicy{Source: true},
|
||||
@@ -436,3 +429,44 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateCSSHref(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
value string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "empty"},
|
||||
{name: "root relative", value: "/assets/report.css"},
|
||||
{name: "relative", value: "assets/report.css"},
|
||||
{name: "parent relative", value: "../assets/report.css"},
|
||||
{name: "query", value: "/assets/report.css?v=20260614"},
|
||||
{name: "http", value: "http://example.com/report.css"},
|
||||
{name: "https", value: "https://example.com/assets/report.css?v=1"},
|
||||
{name: "javascript", value: "javascript:alert(1)", wantErr: true},
|
||||
{name: "data", value: "data:text/css,body{}", wantErr: true},
|
||||
{name: "file", value: "file:///tmp/report.css", wantErr: true},
|
||||
{name: "scheme relative", value: "//example.com/report.css", wantErr: true},
|
||||
{name: "userinfo", value: "https://user@example.com/report.css", wantErr: true},
|
||||
{name: "fragment", value: "/assets/report.css#main", wantErr: true},
|
||||
{name: "space", value: "/assets/report css", wantErr: true},
|
||||
{name: "tab", value: "/assets/report\tcss", wantErr: true},
|
||||
{name: "newline", value: "/assets/report\ncss", wantErr: true},
|
||||
{name: "backslash", value: `assets\report.css`, wantErr: true},
|
||||
{name: "less than", value: "/assets/<report>.css", wantErr: true},
|
||||
{name: "double quote", value: `/assets/"report".css`, wantErr: true},
|
||||
{name: "single quote", value: "/assets/'report'.css", wantErr: true},
|
||||
{name: "query only", value: "?v=1", wantErr: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := validateCSSHref(tt.value)
|
||||
if tt.wantErr && err == nil {
|
||||
t.Fatal("validateCSSHref() error = nil, want error")
|
||||
}
|
||||
if !tt.wantErr && err != nil {
|
||||
t.Fatalf("validateCSSHref() error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
@@ -16,7 +17,9 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
|
||||
switch plan.Action {
|
||||
case ActionSkipSame, ActionSkipDestinationNewer:
|
||||
return nil
|
||||
case ActionPublishNew, ActionReplaceOlder, ActionForceReplace:
|
||||
case ActionPublishNew, ActionUpsertAdditive, ActionReplaceCatalog:
|
||||
return executeCatalog(ctx, req, plan)
|
||||
case ActionReplaceOlder, ActionReplaceConflict, ActionReplaceNewer, ActionReplaceTakeover, ActionForceReplace:
|
||||
if usesSharedRootState(req, plan) {
|
||||
return executeSharedRoot(ctx, req, plan)
|
||||
}
|
||||
@@ -24,11 +27,11 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
|
||||
return fmt.Errorf("cannot execute action %s: %s", plan.Action, plan.Reason)
|
||||
}
|
||||
|
||||
if plan.Action == ActionReplaceOlder {
|
||||
if plan.Action == ActionReplaceOlder || plan.Action == ActionReplaceConflict || plan.Action == ActionReplaceNewer || plan.Action == ActionReplaceTakeover {
|
||||
if plan.ExistingState == nil {
|
||||
return fmt.Errorf("replace requires existing destination state")
|
||||
}
|
||||
if plan.Reconciliation.Mode == config.ReconciliationModeReplace {
|
||||
if plan.Reconciliation.Mode == config.ReconciliationModeReplace || plan.Action == ActionReplaceConflict || plan.Action == ActionReplaceTakeover {
|
||||
if err := req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, state.ManagedOutputPaths(*plan.ExistingState), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true}); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -138,6 +141,152 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func executeCatalog(ctx context.Context, req Request, plan Plan) error {
|
||||
if plan.Action == ActionReplaceCatalog {
|
||||
if plan.ClearDestinationRoot {
|
||||
if err := req.DestinationBackend.DeletePrefix(ctx, req.DestinationBundlePath, storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true}); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureDestinationEmpty(ctx, req.DestinationBackend, req.DestinationBundlePath); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if len(plan.CatalogOutputsToDelete) > 0 {
|
||||
if err := req.DestinationBackend.DeleteManagedOutputs(ctx, req.DestinationBundlePath, catalogOutputPaths(plan.CatalogOutputsToDelete), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
writtenOutputs := make([]Output, 0, len(plan.Outputs))
|
||||
newOutputs := make([]Output, 0, len(plan.Outputs))
|
||||
cleanup := func() {
|
||||
outputs := writtenOutputs
|
||||
if plan.Action == ActionUpsertAdditive {
|
||||
outputs = newOutputs
|
||||
}
|
||||
_ = req.DestinationBackend.DeleteManagedOutputs(ctx, req.DestinationBundlePath, ManagedOutputPaths(outputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true})
|
||||
}
|
||||
for _, output := range plan.Outputs {
|
||||
destinationPath, err := storage.Join(req.DestinationBundlePath, output.DestinationPath)
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
created, err := catalogWriteCreatesOutput(ctx, req.DestinationBackend, destinationPath)
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
data := output.Data
|
||||
if output.Kind == state.OutputKindSource {
|
||||
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, output.SourcePath)
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
data, err = req.SourceBackend.ReadFile(ctx, sourcePath)
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := req.DestinationBackend.WriteFile(ctx, destinationPath, data, storage.WriteOptions{Overwrite: catalogOutputOverwriteAllowed(plan, output), PreferAtomic: true}); err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
writtenOutputs = append(writtenOutputs, output)
|
||||
if created {
|
||||
newOutputs = append(newOutputs, output)
|
||||
}
|
||||
}
|
||||
|
||||
catalogState := catalogStateForPlan(req, plan)
|
||||
if err := state.ValidateCatalog(catalogState); err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
data, err := json.MarshalIndent(catalogState, "", " ")
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
data = append(data, '\n')
|
||||
statePath, err := storage.StatePath(req.DestinationBundlePath)
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
if _, err := req.DestinationBackend.WriteFile(ctx, statePath, data, storage.WriteOptions{Overwrite: catalogStateWriteOverwrites(plan), PreferAtomic: true}); err != nil {
|
||||
cleanup()
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func catalogWriteCreatesOutput(ctx context.Context, backend storage.Backend, destinationPath string) (bool, error) {
|
||||
if _, err := backend.Stat(ctx, destinationPath); err == nil {
|
||||
return false, nil
|
||||
} else if storage.IsNotFound(err) {
|
||||
return true, nil
|
||||
} else {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
|
||||
func catalogOutputOverwriteAllowed(plan Plan, output Output) bool {
|
||||
if plan.ClearDestinationRoot {
|
||||
return false
|
||||
}
|
||||
if plan.SupersededLegacy != nil {
|
||||
return true
|
||||
}
|
||||
if plan.ExistingCatalog == nil {
|
||||
return false
|
||||
}
|
||||
_, ok := state.FindCatalogOutputByPath(plan.ExistingCatalog.Outputs, output.DestinationPath)
|
||||
return ok
|
||||
}
|
||||
|
||||
func catalogStateForPlan(req Request, plan Plan) state.CatalogState {
|
||||
now := requestTime(req)
|
||||
createdAt := now
|
||||
if plan.ExistingCatalog != nil {
|
||||
createdAt = plan.ExistingCatalog.CreatedAt
|
||||
}
|
||||
outputs := make([]state.CatalogOutputFile, 0, len(plan.CatalogOutputsToRetain)+len(plan.CatalogOutputsToWrite))
|
||||
outputs = append(outputs, plan.CatalogOutputsToRetain...)
|
||||
outputs = append(outputs, plan.CatalogOutputsToWrite...)
|
||||
sort.SliceStable(outputs, func(i, j int) bool {
|
||||
if outputs[i].Path != outputs[j].Path {
|
||||
return outputs[i].Path < outputs[j].Path
|
||||
}
|
||||
if outputs[i].PipelineID != outputs[j].PipelineID {
|
||||
return outputs[i].PipelineID < outputs[j].PipelineID
|
||||
}
|
||||
return outputs[i].DestinationID < outputs[j].DestinationID
|
||||
})
|
||||
return state.CatalogState{
|
||||
SchemaVersion: state.CatalogSchemaVersion,
|
||||
DistributorVersion: req.DistributorVersion,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: now,
|
||||
State: state.StatePolicy{Mode: state.StateModeCatalog},
|
||||
Outputs: outputs,
|
||||
}
|
||||
}
|
||||
|
||||
func catalogStateWriteOverwrites(plan Plan) bool {
|
||||
return plan.ExistingCatalog != nil || plan.SupersededLegacy != nil
|
||||
}
|
||||
|
||||
func catalogOutputPaths(outputs []state.CatalogOutputFile) []string {
|
||||
paths := make([]string, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
paths = append(paths, output.Path)
|
||||
}
|
||||
return paths
|
||||
}
|
||||
|
||||
func executeSharedRoot(ctx context.Context, req Request, plan Plan) error {
|
||||
plan.Reconciliation = normalizeReconciliation(plan.Reconciliation)
|
||||
if plan.Action == ActionForceReplace {
|
||||
@@ -148,7 +297,7 @@ func executeSharedRoot(ctx context.Context, req Request, plan Plan) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if plan.Action == ActionReplaceOlder && plan.Reconciliation.Mode == config.ReconciliationModeReplace {
|
||||
if plan.Action == ActionReplaceTakeover || plan.Action == ActionReplaceConflict || (isReconciliationReplacementAction(plan.Action) && plan.Reconciliation.Mode == config.ReconciliationModeReplace) {
|
||||
if err := req.DestinationBackend.DeleteManagedOutputs(ctx, req.DestinationBundlePath, sharedRootOutputPaths(plan.OwnerOutputsToDelete), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true}); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -158,7 +307,7 @@ func executeSharedRoot(ctx context.Context, req Request, plan Plan) error {
|
||||
newOutputs := make([]Output, 0, len(plan.Outputs))
|
||||
cleanup := func() {
|
||||
outputs := writtenOutputs
|
||||
if plan.Reconciliation.Mode == config.ReconciliationModeMerge {
|
||||
if isReconciliationReplacementAction(plan.Action) && plan.Reconciliation.Mode == config.ReconciliationModeMerge {
|
||||
outputs = newOutputs
|
||||
}
|
||||
_ = req.DestinationBackend.DeleteManagedOutputs(ctx, req.DestinationBundlePath, ManagedOutputPaths(outputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true})
|
||||
@@ -240,7 +389,7 @@ func ensureMergeOutputPaths(ctx context.Context, backend storage.Backend, bundle
|
||||
}
|
||||
|
||||
func usesSharedRootState(req Request, plan Plan) bool {
|
||||
return normalizeState(req.State).Mode == config.StateModeSharedRoot || plan.StateMode == config.StateModeSharedRoot
|
||||
return plan.StateMode == config.StateModeSharedRoot
|
||||
}
|
||||
|
||||
func outputManagedByExistingState(output Output, existing *state.DistributorState) bool {
|
||||
@@ -257,6 +406,11 @@ func outputManagedBySharedRootPlan(output Output, plan Plan) bool {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, existing := range plan.TakenOverOwnerOutputs {
|
||||
if existing.Path == output.DestinationPath {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -273,7 +427,7 @@ func stateOutputsForPlan(plan Plan, now time.Time) ([]state.OutputFile, error) {
|
||||
}
|
||||
|
||||
func usesMergeRetention(plan Plan) bool {
|
||||
return plan.Reconciliation.Mode == config.ReconciliationModeMerge && plan.Action == ActionReplaceOlder
|
||||
return plan.Reconciliation.Mode == config.ReconciliationModeMerge && isReconciliationReplacementAction(plan.Action)
|
||||
}
|
||||
|
||||
func sharedRootStateForPlan(req Request, plan Plan, now time.Time) (state.SharedRootState, error) {
|
||||
@@ -283,6 +437,7 @@ func sharedRootStateForPlan(req Request, plan Plan, now time.Time) (state.Shared
|
||||
scope = state.CurrentOwnerScope(req.PipelineID, req.DestinationID)
|
||||
}
|
||||
base := sharedRootBaseState(req, plan, now)
|
||||
base = removeTakenOverSharedRootOutputs(base, plan.TakenOverOwnerOutputs)
|
||||
owner := state.OwnerRecord{
|
||||
Scope: scope,
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: plan.Reconciliation.Mode},
|
||||
@@ -292,7 +447,7 @@ func sharedRootStateForPlan(req Request, plan Plan, now time.Time) (state.Shared
|
||||
owner.Links = &state.LinkState{PrimaryURL: plan.PrimaryURL}
|
||||
}
|
||||
planned := state.ProjectSharedRootOutputs(StateOutputProjections(plan.Outputs), currentOwnerSharedRootOutputs(plan), scope, req.SourceBundle.Manifest, now)
|
||||
if plan.Action == ActionReplaceOlder && plan.Reconciliation.Mode == config.ReconciliationModeMerge {
|
||||
if isReconciliationReplacementAction(plan.Action) && plan.Reconciliation.Mode == config.ReconciliationModeMerge {
|
||||
return state.MergeOwnerOutputs(base, scope, owner, planned)
|
||||
}
|
||||
return state.ReplaceOwnerOutputs(base, scope, owner, planned)
|
||||
@@ -319,6 +474,22 @@ func sharedRootBaseState(req Request, plan Plan, now time.Time) state.SharedRoot
|
||||
return newSharedRootState(req, now)
|
||||
}
|
||||
|
||||
func removeTakenOverSharedRootOutputs(sharedRoot state.SharedRootState, takenOver []state.SharedRootOutputFile) state.SharedRootState {
|
||||
if len(takenOver) == 0 {
|
||||
return sharedRoot
|
||||
}
|
||||
paths := sharedRootOutputPathSet(takenOver)
|
||||
next := sharedRoot
|
||||
next.Outputs = make([]state.SharedRootOutputFile, 0, len(sharedRoot.Outputs))
|
||||
for _, output := range sharedRoot.Outputs {
|
||||
if _, remove := paths[output.Path]; remove {
|
||||
continue
|
||||
}
|
||||
next.Outputs = append(next.Outputs, output)
|
||||
}
|
||||
return next
|
||||
}
|
||||
|
||||
func newSharedRootState(req Request, now time.Time) state.SharedRootState {
|
||||
return state.SharedRootState{
|
||||
SchemaVersion: state.SharedRootSchemaVersion,
|
||||
|
||||
@@ -2,356 +2,167 @@ package publish
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/transform"
|
||||
)
|
||||
|
||||
func TestExecuteCleansUpAfterWriteFailure(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
destinationBackend := &failingBackend{Backend: fake.New(), failPath: "summary.txt"}
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{})
|
||||
req := Request{
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
SourceBundle: sourceBundle,
|
||||
SourceBackend: sourceBackend,
|
||||
DestinationBackend: destinationBackend,
|
||||
DestinationBundlePath: "",
|
||||
Publish: config.PublishPolicy{Source: true},
|
||||
Transfer: config.TransferPolicy{OnDestinationSame: config.TransferActionSkip, OnDestinationOlder: config.TransferActionReplace, OnDestinationNewer: config.TransferActionSkip, OnConflict: config.TransferActionFail},
|
||||
DistributorVersion: "test",
|
||||
func TestExecuteAdditiveWritesOutputsAndCatalog(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
existing := baseCatalog(req)
|
||||
existing.Outputs = []state.CatalogOutputFile{
|
||||
catalogOutput(req, "reports", "archive", "report.md", state.OutputKindSource, planCreatedAt),
|
||||
catalogOutput(req, "reports", "web", "old.txt", state.OutputKindSource, planCreatedAt),
|
||||
}
|
||||
writeCatalogState(t, destinationBackend, "", existing)
|
||||
testutil.WriteFakeFile(t, destinationBackend, "report.md", "old report")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "old.txt", "retained")
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
|
||||
testutil.AssertFakeFile(t, destinationBackend, "report.md", "# Report\nSunny.\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "summary.txt", "Summary\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "old.txt", "retained")
|
||||
catalog := readCatalogState(t, destinationBackend, "")
|
||||
if catalog.SchemaVersion != state.CatalogSchemaVersion || catalog.State.Mode != state.StateModeCatalog {
|
||||
t.Fatalf("catalog identity = schema %d mode %s", catalog.SchemaVersion, catalog.State.Mode)
|
||||
}
|
||||
if len(catalog.Outputs) != 3 {
|
||||
t.Fatalf("catalog outputs = %#v, want three outputs", catalog.Outputs)
|
||||
}
|
||||
report, ok := state.FindCatalogOutputByPath(catalog.Outputs, "report.md")
|
||||
if !ok {
|
||||
t.Fatalf("catalog outputs = %#v, want report.md", catalog.Outputs)
|
||||
}
|
||||
if !report.CreatedAt.Equal(planCreatedAt) || !report.UpdatedAt.Equal(planUpdatedAt) {
|
||||
t.Fatalf("report times = %s/%s, want created preserved and updated now", report.CreatedAt, report.UpdatedAt)
|
||||
}
|
||||
if report.SourcePath != "" {
|
||||
t.Fatalf("source catalog output source_path = %q, want empty", report.SourcePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteReplacementDeletesCurrentOwnerAndPreservesOtherOwners(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowReplacement)
|
||||
existing := baseCatalog(req)
|
||||
existing.Outputs = []state.CatalogOutputFile{
|
||||
catalogOutput(req, "reports", "archive", "report.md", state.OutputKindSource, planCreatedAt),
|
||||
catalogOutput(req, "reports", "archive", "stale.txt", state.OutputKindSource, planCreatedAt),
|
||||
catalogOutput(req, "reports", "web", "shared.txt", state.OutputKindSource, planCreatedAt),
|
||||
}
|
||||
writeCatalogState(t, destinationBackend, "", existing)
|
||||
testutil.WriteFakeFile(t, destinationBackend, "report.md", "old report")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "stale.txt", "delete me")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "shared.txt", "keep me")
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
|
||||
testutil.AssertFakeFile(t, destinationBackend, "report.md", "# Report\nSunny.\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "summary.txt", "Summary\n")
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "stale.txt")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "shared.txt", "keep me")
|
||||
catalog := readCatalogState(t, destinationBackend, "")
|
||||
if _, ok := state.FindCatalogOutputByPath(catalog.Outputs, "stale.txt"); ok {
|
||||
t.Fatalf("catalog outputs = %#v, want stale.txt removed", catalog.Outputs)
|
||||
}
|
||||
if _, ok := state.FindCatalogOutputByPath(catalog.Outputs, "shared.txt"); !ok {
|
||||
t.Fatalf("catalog outputs = %#v, want shared.txt retained", catalog.Outputs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteSupersededReplacementClearsDestinationRootOnly(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowReplacement)
|
||||
req.DestinationBundlePath = "bundle"
|
||||
legacyState := testutil.DestinationState(req.SourceBundle.Manifest, testutil.DestinationStateOptions{})
|
||||
writeJSONState(t, destinationBackend, req.DestinationBundlePath, legacyState)
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle/unplanned.txt", "remove")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "outside.txt", "keep")
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "bundle/unplanned.txt")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\nSunny.\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "outside.txt", "keep")
|
||||
readCatalogState(t, destinationBackend, "bundle")
|
||||
}
|
||||
|
||||
func TestExecuteSupersededAdditiveLeavesUnplannedFilesUnmanaged(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
legacyState := testutil.DestinationState(req.SourceBundle.Manifest, testutil.DestinationStateOptions{})
|
||||
writeJSONState(t, destinationBackend, "", legacyState)
|
||||
testutil.WriteFakeFile(t, destinationBackend, "report.md", "legacy report")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "unplanned.txt", "leave me")
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
|
||||
testutil.AssertFakeFile(t, destinationBackend, "report.md", "# Report\nSunny.\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "unplanned.txt", "leave me")
|
||||
catalog := readCatalogState(t, destinationBackend, "")
|
||||
if _, ok := state.FindCatalogOutputByPath(catalog.Outputs, "unplanned.txt"); ok {
|
||||
t.Fatalf("catalog outputs = %#v, want unplanned file omitted", catalog.Outputs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteFailedWriteDoesNotWriteCatalogState(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := destinationBackend.AddDirectory("report.md"); err != nil {
|
||||
t.Fatalf("add conflicting directory: %v", err)
|
||||
}
|
||||
|
||||
err = Execute(context.Background(), req, plan)
|
||||
if err == nil {
|
||||
t.Fatal("Execute() error = nil, want error")
|
||||
t.Fatal("Execute() error = nil, want write failure")
|
||||
}
|
||||
found, err := destinationBackend.HasAny(context.Background(), "")
|
||||
if err != nil {
|
||||
t.Fatalf("HasAny() error = %v", err)
|
||||
}
|
||||
if found {
|
||||
t.Fatal("destination has content after failed execution")
|
||||
if _, statErr := destinationBackend.Stat(context.Background(), storage.StateFileName); !storage.IsNotFound(statErr) {
|
||||
t.Fatalf("state stat error = %v, want missing state", statErr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteReplaceDeletesOmittedManagedOutputs(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
older := sourceBundle.Manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
older.Files = append([]bundle.ManifestFile(nil), testutil.ValidManifest(testutil.BundleOptions{}).Files...)
|
||||
older.Digest = bundle.BundleDigest(older.Files)
|
||||
testutil.WriteFakeDestinationState(t, destinationBackend, "", older, testutil.DestinationStateOptions{})
|
||||
|
||||
req := testRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionReplaceOlder {
|
||||
t.Fatalf("plan action = %s, want replace_older", plan.Action)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "report.md", "# Report\nNew.\n")
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "summary.txt")
|
||||
destinationState := readFakeState(t, destinationBackend, "")
|
||||
if got, want := len(destinationState.Outputs), 1; got != want {
|
||||
t.Fatalf("state output count = %d, want %d", got, want)
|
||||
}
|
||||
if got, want := destinationState.Reconciliation.Mode, config.ReconciliationModeReplace; got != want {
|
||||
t.Fatalf("reconciliation mode = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteMergeRetainsOmittedAndOverwritesManagedOutputs(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
older := sourceBundle.Manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
older.Files = append([]bundle.ManifestFile(nil), testutil.ValidManifest(testutil.BundleOptions{}).Files...)
|
||||
older.Digest = bundle.BundleDigest(older.Files)
|
||||
existingState := testutil.WriteFakeDestinationState(t, destinationBackend, "", older, testutil.DestinationStateOptions{})
|
||||
|
||||
req := testRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeMerge)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "report.md", "# Report\nNew.\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "summary.txt", "old")
|
||||
destinationState := readFakeState(t, destinationBackend, "")
|
||||
outputs := outputsByPath(destinationState.Outputs)
|
||||
if got, want := len(outputs), 2; got != want {
|
||||
t.Fatalf("state output count = %d, want %d", got, want)
|
||||
}
|
||||
if !outputs["summary.txt"].CreatedAt.Equal(existingState.Outputs[1].CreatedAt) || !outputs["summary.txt"].UpdatedAt.Equal(existingState.Outputs[1].UpdatedAt) {
|
||||
t.Fatalf("retained output timestamps = %#v, want existing %#v", outputs["summary.txt"], existingState.Outputs[1])
|
||||
}
|
||||
if !outputs["report.md"].CreatedAt.Equal(existingState.Outputs[0].CreatedAt) || !outputs["report.md"].UpdatedAt.After(existingState.Outputs[0].UpdatedAt) {
|
||||
t.Fatalf("updated output timestamps = %#v, want preserved created_at and newer updated_at", outputs["report.md"])
|
||||
}
|
||||
if got, want := destinationState.Reconciliation.Mode, config.ReconciliationModeMerge; got != want {
|
||||
t.Fatalf("reconciliation mode = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteMergeFailsOnUnmanagedDestinationPathCollision(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
older := sourceBundle.Manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
older.Files = []bundle.ManifestFile{{
|
||||
Path: "summary.txt",
|
||||
SHA256: bundle.FileDigest([]byte("Summary\n")),
|
||||
Size: int64(len("Summary\n")),
|
||||
}}
|
||||
older.Digest = bundle.BundleDigest(older.Files)
|
||||
testutil.WriteFakeDestinationState(t, destinationBackend, "", older, testutil.DestinationStateOptions{})
|
||||
testutil.WriteFakeFile(t, destinationBackend, "report.md", "unmanaged")
|
||||
|
||||
req := testRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeMerge)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
err = Execute(context.Background(), req, plan)
|
||||
if err == nil {
|
||||
t.Fatal("Execute() error = nil, want unmanaged path collision")
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "report.md", "unmanaged")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "summary.txt", "old")
|
||||
}
|
||||
|
||||
func TestExecuteMergeFailureCleansUpOnlyNewOutputs(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{
|
||||
{Path: "report.md", Data: "# Report\nNew.\n"},
|
||||
{Path: "new.md", Data: "new\n"},
|
||||
{Path: "fail.md", Data: "fail\n"},
|
||||
},
|
||||
})
|
||||
destinationBackend := &failingBackend{Backend: fake.New(), failPath: "fail.md"}
|
||||
older := sourceBundle.Manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
older.Files = []bundle.ManifestFile{sourceBundle.Manifest.Files[0]}
|
||||
older.Digest = bundle.BundleDigest(older.Files)
|
||||
testutil.WriteFakeDestinationState(t, destinationBackend.Backend, "", older, testutil.DestinationStateOptions{})
|
||||
|
||||
req := testRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeMerge)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
err = Execute(context.Background(), req, plan)
|
||||
if err == nil {
|
||||
t.Fatal("Execute() error = nil, want injected failure")
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend.Backend, "report.md", "# Report\nNew.\n")
|
||||
testutil.AssertFakeMissing(t, destinationBackend.Backend, "new.md")
|
||||
testutil.AssertFakeMissing(t, destinationBackend.Backend, "fail.md")
|
||||
}
|
||||
|
||||
func TestExecuteFixedPathSupportsReconciliationModes(t *testing.T) {
|
||||
for _, mode := range []string{config.ReconciliationModeReplace, config.ReconciliationModeMerge} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "new", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
older := sourceBundle.Manifest
|
||||
older.ID = "older.source"
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
testutil.WriteFakeDestinationState(t, destinationBackend, "", older, testutil.DestinationStateOptions{})
|
||||
|
||||
req := testRequest(sourceBackend, destinationBackend, sourceBundle, mode)
|
||||
req.DestinationBundlePath = ""
|
||||
req.PathMapping = config.PathMappingFixed
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionReplaceOlder {
|
||||
t.Fatalf("plan action = %s, want replace_older", plan.Action)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "report.md", "# Report\nNew.\n")
|
||||
destinationState := readFakeState(t, destinationBackend, "")
|
||||
if got, want := destinationState.Reconciliation.Mode, mode; got != want {
|
||||
t.Fatalf("state reconciliation mode = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteReconciliationModesHonorPublishPolicies(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
publish config.PublishPolicy
|
||||
transform config.Transform
|
||||
transformer TransformerResolver
|
||||
wantPaths []string
|
||||
}{
|
||||
{
|
||||
name: "source only",
|
||||
publish: config.PublishPolicy{Source: true},
|
||||
wantPaths: []string{"report.md"},
|
||||
},
|
||||
{
|
||||
name: "generated only",
|
||||
publish: config.PublishPolicy{HTML: true},
|
||||
transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{Enabled: true, Mode: config.TransformModeSidecar}},
|
||||
transformer: testResolver{transform.MarkdownToHTML: testTransformer{outputs: []transform.Output{{
|
||||
Path: "report.html",
|
||||
SourcePath: "report.md",
|
||||
Transform: transform.MarkdownToHTML,
|
||||
Data: []byte("<h1>Report</h1>\n"),
|
||||
SHA256: bundle.FileDigest([]byte("<h1>Report</h1>\n")),
|
||||
Size: int64(len("<h1>Report</h1>\n")),
|
||||
}}}},
|
||||
wantPaths: []string{"report.html"},
|
||||
},
|
||||
{
|
||||
name: "source and generated",
|
||||
publish: config.PublishPolicy{Source: true, HTML: true},
|
||||
transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{Enabled: true, Mode: config.TransformModeSidecar}},
|
||||
transformer: testResolver{transform.MarkdownToHTML: testTransformer{outputs: []transform.Output{{
|
||||
Path: "report.html",
|
||||
SourcePath: "report.md",
|
||||
Transform: transform.MarkdownToHTML,
|
||||
Data: []byte("<h1>Report</h1>\n"),
|
||||
SHA256: bundle.FileDigest([]byte("<h1>Report</h1>\n")),
|
||||
Size: int64(len("<h1>Report</h1>\n")),
|
||||
}}}},
|
||||
wantPaths: []string{"report.md", "report.html"},
|
||||
},
|
||||
}
|
||||
for _, mode := range []string{config.ReconciliationModeReplace, config.ReconciliationModeMerge} {
|
||||
for _, tt := range tests {
|
||||
t.Run(mode+" "+tt.name, func(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
older := sourceBundle.Manifest
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
testutil.WriteFakeDestinationState(t, destinationBackend, "", older, testutil.DestinationStateOptions{})
|
||||
|
||||
req := testRequest(sourceBackend, destinationBackend, sourceBundle, mode)
|
||||
req.Publish = tt.publish
|
||||
req.Transform = tt.transform
|
||||
req.Transformers = tt.transformer
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
destinationState := readFakeState(t, destinationBackend, "")
|
||||
outputs := outputsByPath(destinationState.Outputs)
|
||||
for _, path := range tt.wantPaths {
|
||||
if _, ok := outputs[path]; !ok {
|
||||
t.Fatalf("state outputs = %#v, missing %s", destinationState.Outputs, path)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type failingBackend struct {
|
||||
*fake.Backend
|
||||
failPath string
|
||||
}
|
||||
|
||||
func (b *failingBackend) WriteFile(ctx context.Context, path string, data []byte, opts storage.WriteOptions) (storage.Entry, error) {
|
||||
if path == b.failPath {
|
||||
return storage.Entry{}, fmt.Errorf("injected write failure")
|
||||
}
|
||||
return b.Backend.WriteFile(ctx, path, data, opts)
|
||||
}
|
||||
|
||||
func (b *failingBackend) WriteFrom(ctx context.Context, path string, r io.Reader, opts storage.WriteOptions) (storage.Entry, error) {
|
||||
if path == b.failPath {
|
||||
return storage.Entry{}, fmt.Errorf("injected write failure")
|
||||
}
|
||||
return b.Backend.WriteFrom(ctx, path, r, opts)
|
||||
}
|
||||
|
||||
func testRequest(sourceBackend storage.Backend, destinationBackend storage.Backend, sourceBundle bundle.Bundle, reconciliationMode string) Request {
|
||||
return Request{
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
SourceBundle: sourceBundle,
|
||||
SourceBackend: sourceBackend,
|
||||
DestinationBackend: destinationBackend,
|
||||
DestinationBundlePath: sourceBundle.RootRelativePath,
|
||||
Publish: config.PublishPolicy{Source: true},
|
||||
Reconciliation: config.ReconciliationPolicy{Mode: reconciliationMode},
|
||||
Transfer: config.TransferPolicy{
|
||||
OnDestinationSame: config.TransferActionSkip,
|
||||
OnDestinationOlder: config.TransferActionReplace,
|
||||
OnDestinationNewer: config.TransferActionSkip,
|
||||
OnConflict: config.TransferActionFail,
|
||||
},
|
||||
DistributorVersion: "test",
|
||||
}
|
||||
}
|
||||
|
||||
func readFakeState(t *testing.T, backend storage.Backend, bundlePath string) state.DistributorState {
|
||||
func readCatalogState(t *testing.T, backend *fake.Backend, relative string) state.CatalogState {
|
||||
t.Helper()
|
||||
statePath, err := storage.StatePath(bundlePath)
|
||||
statePath, err := storage.StatePath(relative)
|
||||
if err != nil {
|
||||
t.Fatalf("state path: %v", err)
|
||||
}
|
||||
data, err := backend.ReadFile(context.Background(), statePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read state: %v", err)
|
||||
t.Fatalf("read catalog state: %v", err)
|
||||
}
|
||||
destinationState, err := state.Parse(data)
|
||||
catalog, err := state.ParseCatalog(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse state: %v", err)
|
||||
t.Fatalf("parse catalog state: %v", err)
|
||||
}
|
||||
return destinationState
|
||||
}
|
||||
|
||||
func outputsByPath(outputs []state.OutputFile) map[string]state.OutputFile {
|
||||
byPath := make(map[string]state.OutputFile, len(outputs))
|
||||
for _, output := range outputs {
|
||||
byPath[output.Path] = output
|
||||
}
|
||||
return byPath
|
||||
return catalog
|
||||
}
|
||||
|
||||
@@ -1,193 +0,0 @@
|
||||
package publish
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
|
||||
)
|
||||
|
||||
func TestBuildPlansForcedReplacementOnlyWhenExplicit(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
prepare func(t *testing.T, backend *fake.Backend, source bundle.Manifest)
|
||||
transfer config.TransferPolicy
|
||||
wantReason string
|
||||
forceAction bool
|
||||
}{
|
||||
{
|
||||
name: "unmanaged content",
|
||||
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
|
||||
t.Helper()
|
||||
testutil.WriteFakeFile(t, backend, "bundle/old.txt", "old")
|
||||
},
|
||||
transfer: defaultTransfer(),
|
||||
wantReason: "fail_unmanaged",
|
||||
forceAction: true,
|
||||
},
|
||||
{
|
||||
name: "different source id",
|
||||
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
|
||||
t.Helper()
|
||||
conflict := source
|
||||
conflict.ID = "other.source"
|
||||
testutil.WriteFakeDestinationState(t, backend, "bundle", conflict, testutil.DestinationStateOptions{})
|
||||
},
|
||||
transfer: conflictReplaceTransfer(),
|
||||
wantReason: "requires --force",
|
||||
forceAction: true,
|
||||
},
|
||||
{
|
||||
name: "same created digest conflict",
|
||||
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
|
||||
t.Helper()
|
||||
conflict := testutil.ValidManifest(testutil.BundleOptions{Files: []testutil.SourceFile{{Path: "report.md", Data: "# Different\n"}}})
|
||||
testutil.WriteFakeDestinationState(t, backend, "bundle", conflict, testutil.DestinationStateOptions{})
|
||||
},
|
||||
transfer: conflictReplaceTransfer(),
|
||||
wantReason: "requires --force",
|
||||
forceAction: true,
|
||||
},
|
||||
{
|
||||
name: "pipeline mismatch",
|
||||
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
|
||||
t.Helper()
|
||||
testutil.WriteFakeDestinationState(t, backend, "bundle", source, testutil.DestinationStateOptions{PipelineID: "other-pipeline"})
|
||||
},
|
||||
transfer: conflictReplaceTransfer(),
|
||||
wantReason: "requires --force",
|
||||
forceAction: true,
|
||||
},
|
||||
{
|
||||
name: "destination mismatch",
|
||||
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
|
||||
t.Helper()
|
||||
testutil.WriteFakeDestinationState(t, backend, "bundle", source, testutil.DestinationStateOptions{DestinationID: "other-destination"})
|
||||
},
|
||||
transfer: conflictReplaceTransfer(),
|
||||
wantReason: "requires --force",
|
||||
forceAction: true,
|
||||
},
|
||||
{
|
||||
name: "newer destination",
|
||||
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
|
||||
t.Helper()
|
||||
newer := source
|
||||
newer.Created = newer.Created.AddDate(0, 0, 1)
|
||||
testutil.WriteFakeDestinationState(t, backend, "bundle", newer, testutil.DestinationStateOptions{})
|
||||
},
|
||||
transfer: newerReplaceTransfer(),
|
||||
wantReason: "requires --force",
|
||||
forceAction: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{})
|
||||
destinationBackend := fake.New()
|
||||
tt.prepare(t, destinationBackend, sourceBundle.Manifest)
|
||||
|
||||
req := forceRequest(sourceBackend, destinationBackend, sourceBundle, tt.transfer)
|
||||
_, err := Build(context.Background(), req)
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantReason) {
|
||||
t.Fatalf("Build() error = %v, want %q", err, tt.wantReason)
|
||||
}
|
||||
|
||||
req.Force = true
|
||||
plan, err := Build(context.Background(), req)
|
||||
if tt.forceAction {
|
||||
if err != nil {
|
||||
t.Fatalf("Build() with force error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionForceReplace || !plan.Force {
|
||||
t.Fatalf("forced plan action = %s force=%t", plan.Action, plan.Force)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRequiresConflictPolicyForStateConflicts(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{})
|
||||
destinationBackend := fake.New()
|
||||
conflict := sourceBundle.Manifest
|
||||
conflict.ID = "other.source"
|
||||
testutil.WriteFakeDestinationState(t, destinationBackend, "bundle", conflict, testutil.DestinationStateOptions{})
|
||||
|
||||
req := forceRequest(sourceBackend, destinationBackend, sourceBundle, defaultTransfer())
|
||||
req.Force = true
|
||||
_, err := Build(context.Background(), req)
|
||||
if err == nil || !strings.Contains(err.Error(), "destination source id differs") {
|
||||
t.Fatalf("Build() error = %v, want conservative conflict", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteForcedReplacementDeletesOnlyBundlePath(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{})
|
||||
destinationBackend := fake.New()
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle/old.txt", "old")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle/nested/old.txt", "old")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "outside.txt", "outside")
|
||||
|
||||
req := forceRequest(sourceBackend, destinationBackend, sourceBundle, defaultTransfer())
|
||||
req.Force = true
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionForceReplace {
|
||||
t.Fatalf("plan action = %s, want force_replace", plan.Action)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\nSunny.\n")
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "bundle/old.txt")
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "bundle/nested/old.txt")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "outside.txt", "outside")
|
||||
}
|
||||
|
||||
func forceRequest(sourceBackend, destinationBackend *fake.Backend, sourceBundle bundle.Bundle, transfer config.TransferPolicy) Request {
|
||||
return Request{
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
SourceBundle: sourceBundle,
|
||||
SourceBackend: sourceBackend,
|
||||
DestinationBackend: destinationBackend,
|
||||
DestinationBundlePath: sourceBundle.RootRelativePath,
|
||||
Publish: config.PublishPolicy{Source: true},
|
||||
Transfer: transfer,
|
||||
DistributorVersion: "test",
|
||||
}
|
||||
}
|
||||
|
||||
func defaultTransfer() config.TransferPolicy {
|
||||
return config.TransferPolicy{
|
||||
OnDestinationSame: config.TransferActionSkip,
|
||||
OnDestinationOlder: config.TransferActionReplace,
|
||||
OnDestinationNewer: config.TransferActionSkip,
|
||||
OnConflict: config.TransferActionFail,
|
||||
}
|
||||
}
|
||||
|
||||
func conflictReplaceTransfer() config.TransferPolicy {
|
||||
transfer := defaultTransfer()
|
||||
transfer.OnConflict = config.TransferActionReplace
|
||||
return transfer
|
||||
}
|
||||
|
||||
func newerReplaceTransfer() config.TransferPolicy {
|
||||
transfer := defaultTransfer()
|
||||
transfer.OnDestinationNewer = config.TransferActionReplace
|
||||
return transfer
|
||||
}
|
||||
@@ -29,6 +29,7 @@ func PlanOutputs(ctx context.Context, req Request) ([]Output, error) {
|
||||
Markdown: transform.MarkdownOptions{
|
||||
Mode: req.Transform.MarkdownToHTML.Mode,
|
||||
Input: req.Transform.MarkdownToHTML.Input,
|
||||
CssHref: req.Transform.MarkdownToHTML.CssHref,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
|
||||
@@ -167,6 +167,7 @@ func TestPlanOutputsPassesMarkdownOptions(t *testing.T) {
|
||||
Enabled: true,
|
||||
Mode: config.TransformModeIndex,
|
||||
Input: "report.md",
|
||||
CssHref: "/assets/report.css",
|
||||
}},
|
||||
Transformers: testResolver{transform.MarkdownToHTML: transformer},
|
||||
})
|
||||
@@ -174,8 +175,8 @@ func TestPlanOutputsPassesMarkdownOptions(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("PlanOutputs() error = %v", err)
|
||||
}
|
||||
if transformer.request.Markdown.Mode != config.TransformModeIndex || transformer.request.Markdown.Input != "report.md" {
|
||||
t.Fatalf("markdown options = %#v, want index/report.md", transformer.request.Markdown)
|
||||
if transformer.request.Markdown.Mode != config.TransformModeIndex || transformer.request.Markdown.Input != "report.md" || transformer.request.Markdown.CssHref != "/assets/report.css" {
|
||||
t.Fatalf("markdown options = %#v, want index/report.md with css href", transformer.request.Markdown)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -193,12 +194,6 @@ func TestBuildRejectsHTMLWithoutTransform(t *testing.T) {
|
||||
DestinationBundlePath: "",
|
||||
SourceBundle: sourceBundle,
|
||||
Publish: config.PublishPolicy{HTML: true},
|
||||
Transfer: config.TransferPolicy{
|
||||
OnDestinationSame: config.TransferActionSkip,
|
||||
OnDestinationOlder: config.TransferActionReplace,
|
||||
OnDestinationNewer: config.TransferActionSkip,
|
||||
OnConflict: config.TransferActionFail,
|
||||
},
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want missing transform error")
|
||||
|
||||
@@ -3,6 +3,7 @@ package publish
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
@@ -16,11 +17,16 @@ type Action string
|
||||
const (
|
||||
ActionPublishNew Action = "publish_new"
|
||||
ActionReplaceOlder Action = "replace_older"
|
||||
ActionReplaceConflict Action = "replace_conflict"
|
||||
ActionReplaceNewer Action = "replace_newer"
|
||||
ActionSkipSame Action = "skip_same"
|
||||
ActionSkipDestinationNewer Action = "skip_destination_newer"
|
||||
ActionFailConflict Action = "fail_conflict"
|
||||
ActionFailUnmanaged Action = "fail_unmanaged"
|
||||
ActionForceReplace Action = "force_replace"
|
||||
ActionReplaceTakeover Action = "replace_takeover"
|
||||
ActionUpsertAdditive Action = "upsert_additive"
|
||||
ActionReplaceCatalog Action = "replace_catalog"
|
||||
)
|
||||
|
||||
type Request struct {
|
||||
@@ -34,18 +40,28 @@ type Request struct {
|
||||
Publish config.PublishPolicy
|
||||
Transform config.Transform
|
||||
Links *config.Links
|
||||
State config.StatePolicy
|
||||
Reconciliation config.ReconciliationPolicy
|
||||
Workflow string
|
||||
Transformers TransformerResolver
|
||||
Transfer config.TransferPolicy
|
||||
DistributorVersion string
|
||||
Force bool
|
||||
Now time.Time
|
||||
}
|
||||
|
||||
type TransformerResolver interface {
|
||||
Get(name string) (transform.Transformer, bool)
|
||||
}
|
||||
|
||||
type Output struct {
|
||||
SourcePath string
|
||||
DestinationPath string
|
||||
Kind string
|
||||
Transform string
|
||||
URL string
|
||||
Data []byte
|
||||
SHA256 string
|
||||
Size int64
|
||||
}
|
||||
|
||||
type Plan struct {
|
||||
PipelineID string
|
||||
DestinationID string
|
||||
@@ -57,27 +73,36 @@ type Plan struct {
|
||||
Reason string
|
||||
Force bool
|
||||
PrimaryURL string
|
||||
StateMode string
|
||||
Workflow string
|
||||
OwnerScope state.OwnerScope
|
||||
Reconciliation config.ReconciliationPolicy
|
||||
Outputs []Output
|
||||
ExistingCatalog *state.CatalogState
|
||||
SupersededLegacy *state.SupersededLegacyState
|
||||
CatalogOutputsToWrite []state.CatalogOutputFile
|
||||
CatalogOutputsToRetain []state.CatalogOutputFile
|
||||
CatalogOutputsToDelete []state.CatalogOutputFile
|
||||
ClearDestinationRoot bool
|
||||
|
||||
// Retained while the executor is migrated to catalog state.
|
||||
StateMode string
|
||||
Reconciliation config.ReconciliationPolicy
|
||||
TakeoverMode string
|
||||
ExistingState *state.DistributorState
|
||||
ExistingSharedRoot *state.SharedRootState
|
||||
OtherOwnerOutputs []state.SharedRootOutputFile
|
||||
TakenOverOwnerOutputs []state.SharedRootOutputFile
|
||||
RetainedOwnerOutputs []state.SharedRootOutputFile
|
||||
OwnerOutputsToDelete []state.SharedRootOutputFile
|
||||
OwnerOutputsToWrite []Output
|
||||
}
|
||||
|
||||
type Output struct {
|
||||
SourcePath string
|
||||
DestinationPath string
|
||||
Kind string
|
||||
Transform string
|
||||
URL string
|
||||
Data []byte
|
||||
SHA256 string
|
||||
Size int64
|
||||
type catalogPlanDetails struct {
|
||||
Action Action
|
||||
Reason string
|
||||
CatalogOutputsToWrite []state.CatalogOutputFile
|
||||
CatalogOutputsToRetain []state.CatalogOutputFile
|
||||
CatalogOutputsToDelete []state.CatalogOutputFile
|
||||
ClearDestinationRoot bool
|
||||
}
|
||||
|
||||
func Build(ctx context.Context, req Request) (Plan, error) {
|
||||
@@ -96,10 +121,9 @@ func Build(ctx context.Context, req Request) (Plan, error) {
|
||||
if err != nil {
|
||||
return Plan{}, err
|
||||
}
|
||||
comparison := compareDestination(req, status)
|
||||
action, reason := actionForComparison(comparison, req.Transfer, req.Force)
|
||||
reconciliation := normalizeReconciliation(req.Reconciliation)
|
||||
stateMode := normalizeState(req.State).Mode
|
||||
workflow := normalizeWorkflow(req.Workflow)
|
||||
scope := state.CurrentOwnerScope(req.PipelineID, req.DestinationID)
|
||||
now := requestTime(req)
|
||||
plan := Plan{
|
||||
PipelineID: req.PipelineID,
|
||||
DestinationID: req.DestinationID,
|
||||
@@ -107,32 +131,37 @@ func Build(ctx context.Context, req Request) (Plan, error) {
|
||||
BundlePath: req.SourceBundle.RootRelativePath,
|
||||
DestinationBundlePath: req.DestinationBundlePath,
|
||||
PathMapping: req.PathMapping,
|
||||
Action: action,
|
||||
Reason: reason,
|
||||
Force: action == ActionForceReplace,
|
||||
PrimaryURL: primaryURL,
|
||||
StateMode: stateMode,
|
||||
OwnerScope: state.CurrentOwnerScope(req.PipelineID, req.DestinationID),
|
||||
Reconciliation: reconciliation,
|
||||
Workflow: workflow,
|
||||
OwnerScope: scope,
|
||||
Outputs: outputs,
|
||||
ExistingState: status.State,
|
||||
ExistingSharedRoot: status.SharedRoot,
|
||||
ExistingCatalog: status.Catalog,
|
||||
SupersededLegacy: status.SupersededLegacy,
|
||||
}
|
||||
if stateMode == config.StateModeSharedRoot {
|
||||
sharedDetails, err := planSharedRootOwner(ctx, req, status, action, reconciliation, outputs)
|
||||
plan.OtherOwnerOutputs = sharedDetails.OtherOwnerOutputs
|
||||
plan.RetainedOwnerOutputs = sharedDetails.RetainedOwnerOutputs
|
||||
plan.OwnerOutputsToDelete = sharedDetails.OwnerOutputsToDelete
|
||||
plan.OwnerOutputsToWrite = sharedDetails.OwnerOutputsToWrite
|
||||
if status.StateErr != nil {
|
||||
plan.Action = ActionFailConflict
|
||||
plan.Reason = status.StateErr.Error()
|
||||
return plan, fmt.Errorf("%s: %s", plan.Action, plan.Reason)
|
||||
}
|
||||
|
||||
var details catalogPlanDetails
|
||||
switch {
|
||||
case status.Catalog != nil:
|
||||
details, err = planExistingCatalog(ctx, req, *status.Catalog, outputs, workflow, scope, now)
|
||||
case status.SupersededLegacy != nil:
|
||||
details = planSupersededLegacy(req, outputs, workflow, scope, now)
|
||||
default:
|
||||
details, err = planWithoutCatalog(ctx, req, outputs, workflow, scope, now)
|
||||
}
|
||||
plan.Action = details.Action
|
||||
plan.Reason = details.Reason
|
||||
plan.CatalogOutputsToWrite = details.CatalogOutputsToWrite
|
||||
plan.CatalogOutputsToRetain = details.CatalogOutputsToRetain
|
||||
plan.CatalogOutputsToDelete = details.CatalogOutputsToDelete
|
||||
plan.ClearDestinationRoot = details.ClearDestinationRoot
|
||||
if err != nil {
|
||||
plan.Action = sharedDetails.Action
|
||||
plan.Reason = sharedDetails.Reason
|
||||
return plan, err
|
||||
}
|
||||
}
|
||||
if action == ActionFailConflict || action == ActionFailUnmanaged {
|
||||
return plan, fmt.Errorf("%s: %s", action, reason)
|
||||
}
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
@@ -152,139 +181,134 @@ func validateRequest(req Request) error {
|
||||
if err := config.ValidatePublishTransformPolicy(req.Publish, req.Transform); err != nil {
|
||||
return fmt.Errorf("publish/transform policy: %w", err)
|
||||
}
|
||||
switch normalizeReconciliation(req.Reconciliation).Mode {
|
||||
case config.ReconciliationModeReplace, config.ReconciliationModeMerge:
|
||||
switch normalizeWorkflow(req.Workflow) {
|
||||
case config.WorkflowAdditive, config.WorkflowReplacement:
|
||||
default:
|
||||
return fmt.Errorf("reconciliation.mode must be %s or %s", config.ReconciliationModeReplace, config.ReconciliationModeMerge)
|
||||
}
|
||||
switch normalizeState(req.State).Mode {
|
||||
case config.StateModeSingleOwner, config.StateModeSharedRoot:
|
||||
default:
|
||||
return fmt.Errorf("state.mode must be %s or %s", config.StateModeSingleOwner, config.StateModeSharedRoot)
|
||||
return fmt.Errorf("destination.workflow must be %s or %s", config.WorkflowAdditive, config.WorkflowReplacement)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeReconciliation(policy config.ReconciliationPolicy) config.ReconciliationPolicy {
|
||||
if policy.Mode == "" {
|
||||
policy.Mode = config.ReconciliationModeReplace
|
||||
func normalizeWorkflow(workflow string) string {
|
||||
if workflow == "" {
|
||||
return config.WorkflowAdditive
|
||||
}
|
||||
return policy
|
||||
return workflow
|
||||
}
|
||||
|
||||
func normalizeState(policy config.StatePolicy) config.StatePolicy {
|
||||
if policy.Mode == "" {
|
||||
policy.Mode = config.StateModeSingleOwner
|
||||
func requestTime(req Request) time.Time {
|
||||
if req.Now.IsZero() {
|
||||
return time.Now().UTC()
|
||||
}
|
||||
return policy
|
||||
return req.Now.UTC()
|
||||
}
|
||||
|
||||
func compareDestination(req Request, status state.DestinationStatus) state.Comparison {
|
||||
if normalizeState(req.State).Mode == config.StateModeSharedRoot {
|
||||
scope := state.CurrentOwnerScope(req.PipelineID, req.DestinationID)
|
||||
comparison := state.CompareSharedRootOwner(req.SourceBundle.Manifest, scope, status)
|
||||
if req.PathMapping != config.PathMappingFixed || comparison.Outcome != state.OutcomeDifferentSourceConflict {
|
||||
return comparison
|
||||
func planExistingCatalog(ctx context.Context, req Request, catalog state.CatalogState, outputs []Output, workflow string, scope state.OwnerScope, now time.Time) (catalogPlanDetails, error) {
|
||||
if err := rejectCatalogUnmanagedCollisions(ctx, req.DestinationBackend, req.DestinationBundlePath, catalog.Outputs, outputs); err != nil {
|
||||
return catalogPlanDetails{
|
||||
Action: ActionFailUnmanaged,
|
||||
Reason: err.Error(),
|
||||
}, fmt.Errorf("%s: %s", ActionFailUnmanaged, err)
|
||||
}
|
||||
destinationManifest, ok := sharedRootComparisonManifest(status, scope)
|
||||
if !ok {
|
||||
return comparison
|
||||
planned := outputPathSet(outputs)
|
||||
details := catalogPlanDetails{
|
||||
Action: actionForWorkflow(workflow),
|
||||
CatalogOutputsToWrite: catalogOutputsForPlan(req, outputs, catalog.Outputs, scope, now),
|
||||
}
|
||||
if destinationManifest.Created.Before(req.SourceBundle.Manifest.Created) {
|
||||
return state.Comparison{Outcome: state.OutcomeDestinationOlder, Reason: "fixed destination source is older than selected source"}
|
||||
}
|
||||
if destinationManifest.Created.After(req.SourceBundle.Manifest.Created) {
|
||||
return state.Comparison{Outcome: state.OutcomeDestinationNewer, Reason: "fixed destination source is newer than selected source"}
|
||||
}
|
||||
return comparison
|
||||
}
|
||||
comparison := state.Compare(req.SourceBundle.Manifest, req.PipelineID, req.DestinationID, status)
|
||||
if req.PathMapping != config.PathMappingFixed || comparison.Outcome != state.OutcomeDifferentSourceConflict || status.State == nil {
|
||||
return comparison
|
||||
}
|
||||
destinationManifest := status.State.Source.Manifest
|
||||
if destinationManifest.Created.Before(req.SourceBundle.Manifest.Created) {
|
||||
return state.Comparison{Outcome: state.OutcomeDestinationOlder, Reason: "fixed destination source is older than selected source"}
|
||||
}
|
||||
if destinationManifest.Created.After(req.SourceBundle.Manifest.Created) {
|
||||
return state.Comparison{Outcome: state.OutcomeDestinationNewer, Reason: "fixed destination source is newer than selected source"}
|
||||
}
|
||||
return comparison
|
||||
}
|
||||
|
||||
func sharedRootComparisonManifest(status state.DestinationStatus, scope state.OwnerScope) (bundle.Manifest, bool) {
|
||||
if status.SharedRoot != nil {
|
||||
return status.SharedRoot.SourceManifest(scope)
|
||||
}
|
||||
if status.State != nil && status.State.PipelineID == scope.PipelineID && status.State.DestinationID == scope.DestinationID {
|
||||
return status.State.Source.Manifest, true
|
||||
}
|
||||
return bundle.Manifest{}, false
|
||||
}
|
||||
|
||||
type sharedRootPlanDetails struct {
|
||||
Action Action
|
||||
Reason string
|
||||
OtherOwnerOutputs []state.SharedRootOutputFile
|
||||
RetainedOwnerOutputs []state.SharedRootOutputFile
|
||||
OwnerOutputsToDelete []state.SharedRootOutputFile
|
||||
OwnerOutputsToWrite []Output
|
||||
}
|
||||
|
||||
func planSharedRootOwner(ctx context.Context, req Request, status state.DestinationStatus, action Action, reconciliation config.ReconciliationPolicy, outputs []Output) (sharedRootPlanDetails, error) {
|
||||
scope := state.CurrentOwnerScope(req.PipelineID, req.DestinationID)
|
||||
details := sharedRootPlanDetails{Action: action}
|
||||
if !isWriteAction(action) {
|
||||
details.OtherOwnerOutputs = otherOwnerOutputs(status, scope)
|
||||
return details, nil
|
||||
}
|
||||
|
||||
plannedPaths := outputPaths(outputs)
|
||||
if action == ActionForceReplace {
|
||||
details.OwnerOutputsToWrite = append([]Output(nil), outputs...)
|
||||
return details, nil
|
||||
}
|
||||
if conflict, ok := sharedRootPathOwnershipConflict(status, scope, plannedPaths); ok {
|
||||
reason := fmt.Sprintf("destination output path %s is owned by %s/%s", conflict.Path, conflict.Owner.PipelineID, conflict.Owner.DestinationID)
|
||||
details.Action = ActionFailConflict
|
||||
details.Reason = reason
|
||||
return details, fmt.Errorf("%s: %s", ActionFailConflict, reason)
|
||||
}
|
||||
if err := rejectSharedRootUnmanagedCollisions(ctx, req.DestinationBackend, req.DestinationBundlePath, status, scope, plannedPaths); err != nil {
|
||||
details.Action = ActionFailUnmanaged
|
||||
details.Reason = err.Error()
|
||||
return details, fmt.Errorf("%s: %s", ActionFailUnmanaged, err)
|
||||
}
|
||||
|
||||
details.OtherOwnerOutputs = otherOwnerOutputs(status, scope)
|
||||
ownerOutputs := currentOwnerOutputs(status, scope)
|
||||
planned := make(map[string]struct{}, len(plannedPaths))
|
||||
for _, path := range plannedPaths {
|
||||
planned[path] = struct{}{}
|
||||
}
|
||||
for _, output := range ownerOutputs {
|
||||
for _, output := range catalog.Outputs {
|
||||
if _, exists := planned[output.Path]; exists {
|
||||
continue
|
||||
}
|
||||
if action == ActionReplaceOlder && reconciliation.Mode == config.ReconciliationModeReplace {
|
||||
details.OwnerOutputsToDelete = append(details.OwnerOutputsToDelete, output)
|
||||
if workflow == config.WorkflowReplacement && output.PipelineID == scope.PipelineID && output.DestinationID == scope.DestinationID {
|
||||
details.CatalogOutputsToDelete = append(details.CatalogOutputsToDelete, output)
|
||||
continue
|
||||
}
|
||||
if action == ActionReplaceOlder && reconciliation.Mode == config.ReconciliationModeMerge {
|
||||
details.RetainedOwnerOutputs = append(details.RetainedOwnerOutputs, output)
|
||||
details.CatalogOutputsToRetain = append(details.CatalogOutputsToRetain, output)
|
||||
}
|
||||
}
|
||||
details.OwnerOutputsToWrite = append([]Output(nil), outputs...)
|
||||
return details, nil
|
||||
}
|
||||
|
||||
func isWriteAction(action Action) bool {
|
||||
switch action {
|
||||
case ActionPublishNew, ActionReplaceOlder, ActionForceReplace:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
func planSupersededLegacy(req Request, outputs []Output, workflow string, scope state.OwnerScope, now time.Time) catalogPlanDetails {
|
||||
details := catalogPlanDetails{
|
||||
Action: actionForWorkflow(workflow),
|
||||
CatalogOutputsToWrite: catalogOutputsForPlan(req, outputs, nil, scope, now),
|
||||
}
|
||||
if workflow == config.WorkflowReplacement {
|
||||
details.ClearDestinationRoot = true
|
||||
}
|
||||
return details
|
||||
}
|
||||
|
||||
func planWithoutCatalog(ctx context.Context, req Request, outputs []Output, workflow string, scope state.OwnerScope, now time.Time) (catalogPlanDetails, error) {
|
||||
if err := rejectCatalogUnmanagedCollisions(ctx, req.DestinationBackend, req.DestinationBundlePath, nil, outputs); err != nil {
|
||||
return catalogPlanDetails{
|
||||
Action: ActionFailUnmanaged,
|
||||
Reason: err.Error(),
|
||||
}, fmt.Errorf("%s: %s", ActionFailUnmanaged, err)
|
||||
}
|
||||
return catalogPlanDetails{
|
||||
Action: ActionPublishNew,
|
||||
CatalogOutputsToWrite: catalogOutputsForPlan(req, outputs, nil, scope, now),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func actionForWorkflow(workflow string) Action {
|
||||
if workflow == config.WorkflowReplacement {
|
||||
return ActionReplaceCatalog
|
||||
}
|
||||
return ActionUpsertAdditive
|
||||
}
|
||||
|
||||
func catalogOutputsForPlan(req Request, outputs []Output, existing []state.CatalogOutputFile, scope state.OwnerScope, now time.Time) []state.CatalogOutputFile {
|
||||
files := make([]state.CatalogOutputFile, 0, len(outputs))
|
||||
source := state.CatalogSourceIdentity{
|
||||
ID: req.SourceBundle.Manifest.ID,
|
||||
Digest: req.SourceBundle.Manifest.Digest,
|
||||
Created: req.SourceBundle.Manifest.Created,
|
||||
}
|
||||
for _, output := range outputs {
|
||||
createdAt := now
|
||||
if existingOutput, ok := state.FindCatalogOutputByPath(existing, output.DestinationPath); ok {
|
||||
createdAt = existingOutput.CreatedAt
|
||||
}
|
||||
file := state.CatalogOutputFile{
|
||||
Path: output.DestinationPath,
|
||||
PipelineID: scope.PipelineID,
|
||||
DestinationID: scope.DestinationID,
|
||||
Source: source,
|
||||
Kind: output.Kind,
|
||||
URL: output.URL,
|
||||
SHA256: output.SHA256,
|
||||
Size: output.Size,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: now,
|
||||
}
|
||||
if output.Kind == state.OutputKindGenerated {
|
||||
file.SourcePath = output.SourcePath
|
||||
file.Transform = output.Transform
|
||||
}
|
||||
files = append(files, file)
|
||||
}
|
||||
return files
|
||||
}
|
||||
|
||||
func rejectCatalogUnmanagedCollisions(ctx context.Context, backend storage.Backend, bundlePath string, existing []state.CatalogOutputFile, outputs []Output) error {
|
||||
managed := catalogOutputPathSet(existing)
|
||||
for _, output := range outputs {
|
||||
if _, exists := managed[output.DestinationPath]; exists {
|
||||
continue
|
||||
}
|
||||
destinationPath, err := storage.Join(bundlePath, output.DestinationPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := backend.Stat(ctx, destinationPath); err == nil {
|
||||
return fmt.Errorf("destination output path %s exists but is not managed by catalog state", storage.DisplayPath(output.DestinationPath))
|
||||
} else if !storage.IsNotFound(err) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func outputPaths(outputs []Output) []string {
|
||||
@@ -295,131 +319,37 @@ func outputPaths(outputs []Output) []string {
|
||||
return paths
|
||||
}
|
||||
|
||||
func sharedRootPathOwnershipConflict(status state.DestinationStatus, scope state.OwnerScope, paths []string) (state.PathOwnershipConflict, bool) {
|
||||
if status.SharedRoot != nil {
|
||||
return status.SharedRoot.PathOwnershipConflict(scope, paths)
|
||||
func outputPathSet(outputs []Output) map[string]struct{} {
|
||||
paths := make(map[string]struct{}, len(outputs))
|
||||
for _, output := range outputs {
|
||||
paths[output.DestinationPath] = struct{}{}
|
||||
}
|
||||
return state.PathOwnershipConflict{}, false
|
||||
return paths
|
||||
}
|
||||
|
||||
func rejectSharedRootUnmanagedCollisions(ctx context.Context, backend storage.Backend, bundlePath string, status state.DestinationStatus, scope state.OwnerScope, paths []string) error {
|
||||
for _, path := range paths {
|
||||
if pathManagedBySharedRootStatus(status, scope, path) {
|
||||
continue
|
||||
func catalogOutputPathSet(outputs []state.CatalogOutputFile) map[string]struct{} {
|
||||
paths := make(map[string]struct{}, len(outputs))
|
||||
for _, output := range outputs {
|
||||
paths[output.Path] = struct{}{}
|
||||
}
|
||||
destinationPath, err := storage.Join(bundlePath, path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := backend.Stat(ctx, destinationPath); err == nil {
|
||||
return fmt.Errorf("destination output path %s exists but is not managed by destination state", storage.DisplayPath(path))
|
||||
} else if !storage.IsNotFound(err) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
return paths
|
||||
}
|
||||
|
||||
func pathManagedBySharedRootStatus(status state.DestinationStatus, scope state.OwnerScope, path string) bool {
|
||||
if status.SharedRoot != nil {
|
||||
_, exists := status.SharedRoot.OutputOwner(path)
|
||||
return exists
|
||||
func normalizeReconciliation(policy config.ReconciliationPolicy) config.ReconciliationPolicy {
|
||||
if policy.Mode == "" {
|
||||
policy.Mode = config.ReconciliationModeReplace
|
||||
}
|
||||
if status.State != nil && status.State.PipelineID == scope.PipelineID && status.State.DestinationID == scope.DestinationID {
|
||||
_, exists := state.FindOutputByPath(status.State.Outputs, path)
|
||||
return exists
|
||||
}
|
||||
return false
|
||||
return policy
|
||||
}
|
||||
|
||||
func otherOwnerOutputs(status state.DestinationStatus, scope state.OwnerScope) []state.SharedRootOutputFile {
|
||||
if status.SharedRoot == nil {
|
||||
return nil
|
||||
}
|
||||
outputs := make([]state.SharedRootOutputFile, 0, len(status.SharedRoot.Outputs))
|
||||
for _, output := range status.SharedRoot.Outputs {
|
||||
if output.Owner != scope {
|
||||
outputs = append(outputs, output)
|
||||
}
|
||||
}
|
||||
return outputs
|
||||
func isReconciliationReplacementAction(action Action) bool {
|
||||
return action == ActionReplaceOlder || action == ActionReplaceNewer
|
||||
}
|
||||
|
||||
func currentOwnerOutputs(status state.DestinationStatus, scope state.OwnerScope) []state.SharedRootOutputFile {
|
||||
if status.SharedRoot != nil {
|
||||
outputs := make([]state.SharedRootOutputFile, 0, len(status.SharedRoot.Outputs))
|
||||
for _, output := range status.SharedRoot.Outputs {
|
||||
if output.Owner == scope {
|
||||
outputs = append(outputs, output)
|
||||
}
|
||||
}
|
||||
return outputs
|
||||
}
|
||||
if status.State != nil && status.State.PipelineID == scope.PipelineID && status.State.DestinationID == scope.DestinationID {
|
||||
outputs := make([]state.SharedRootOutputFile, 0, len(status.State.Outputs))
|
||||
for _, output := range status.State.Outputs {
|
||||
outputs = append(outputs, state.SharedRootOutputFile{
|
||||
Path: output.Path,
|
||||
Kind: output.Kind,
|
||||
SourcePath: output.SourcePath,
|
||||
Transform: output.Transform,
|
||||
URL: output.URL,
|
||||
SHA256: output.SHA256,
|
||||
Size: output.Size,
|
||||
Owner: scope,
|
||||
SourceID: status.State.Source.Manifest.ID,
|
||||
SourceDigest: status.State.Source.Manifest.Digest,
|
||||
SourceCreated: status.State.Source.Manifest.Created,
|
||||
CreatedAt: output.CreatedAt,
|
||||
UpdatedAt: output.UpdatedAt,
|
||||
})
|
||||
}
|
||||
return outputs
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func actionForComparison(comparison state.Comparison, transfer config.TransferPolicy, force bool) (Action, string) {
|
||||
switch comparison.Outcome {
|
||||
case state.OutcomeDestinationAbsent:
|
||||
return ActionPublishNew, comparison.Reason
|
||||
case state.OutcomeDestinationUnmanaged:
|
||||
if force {
|
||||
return ActionForceReplace, "forced replacement of unmanaged destination content"
|
||||
}
|
||||
return ActionFailUnmanaged, comparison.Reason
|
||||
case state.OutcomeInvalidState:
|
||||
return ActionFailConflict, comparison.Reason
|
||||
case state.OutcomeIdentityMismatch, state.OutcomeSameCreatedConflict, state.OutcomeDifferentSourceConflict:
|
||||
if transfer.OnConflict == config.TransferActionReplace {
|
||||
if force {
|
||||
return ActionForceReplace, "forced replacement of conflicting destination state: " + comparison.Reason
|
||||
}
|
||||
return ActionFailConflict, "destination conflict replacement requires --force"
|
||||
}
|
||||
return ActionFailConflict, comparison.Reason
|
||||
case state.OutcomeSameSource:
|
||||
if transfer.OnDestinationSame == config.TransferActionFail {
|
||||
return ActionFailConflict, "destination matches source and transfer policy requires failure"
|
||||
}
|
||||
return ActionSkipSame, comparison.Reason
|
||||
case state.OutcomeDestinationOlder:
|
||||
if transfer.OnDestinationOlder == config.TransferActionFail {
|
||||
return ActionFailConflict, "destination is older and transfer policy requires failure"
|
||||
}
|
||||
return ActionReplaceOlder, comparison.Reason
|
||||
case state.OutcomeDestinationNewer:
|
||||
if transfer.OnDestinationNewer == config.TransferActionReplace {
|
||||
if force {
|
||||
return ActionForceReplace, "forced replacement of newer destination state"
|
||||
}
|
||||
return ActionFailConflict, "destination is newer and replacement requires --force"
|
||||
}
|
||||
if transfer.OnDestinationNewer == config.TransferActionFail {
|
||||
return ActionFailConflict, "destination is newer and transfer policy requires failure"
|
||||
}
|
||||
return ActionSkipDestinationNewer, comparison.Reason
|
||||
default:
|
||||
return ActionFailConflict, "unsupported comparison outcome"
|
||||
func sharedRootOutputPathSet(outputs []state.SharedRootOutputFile) map[string]struct{} {
|
||||
paths := make(map[string]struct{}, len(outputs))
|
||||
for _, output := range outputs {
|
||||
paths[output.Path] = struct{}{}
|
||||
}
|
||||
return paths
|
||||
}
|
||||
|
||||
296
internal/publish/plan_test.go
Normal file
296
internal/publish/plan_test.go
Normal file
@@ -0,0 +1,296 @@
|
||||
package publish
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
|
||||
)
|
||||
|
||||
var (
|
||||
planCreatedAt = time.Date(2026, 5, 30, 11, 12, 0, 0, time.UTC)
|
||||
planUpdatedAt = time.Date(2026, 6, 1, 9, 30, 0, 0, time.UTC)
|
||||
)
|
||||
|
||||
func TestBuildAdditivePublishesCatalogOutputs(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionPublishNew || plan.Workflow != config.WorkflowAdditive {
|
||||
t.Fatalf("plan action=%s workflow=%s, want publish_new additive", plan.Action, plan.Workflow)
|
||||
}
|
||||
if len(plan.CatalogOutputsToWrite) != 2 || len(plan.CatalogOutputsToRetain) != 0 || len(plan.CatalogOutputsToDelete) != 0 {
|
||||
t.Fatalf("catalog write=%d retain=%d delete=%d", len(plan.CatalogOutputsToWrite), len(plan.CatalogOutputsToRetain), len(plan.CatalogOutputsToDelete))
|
||||
}
|
||||
for _, output := range plan.CatalogOutputsToWrite {
|
||||
if output.PipelineID != "reports" || output.DestinationID != "archive" {
|
||||
t.Fatalf("catalog output owner = %s/%s", output.PipelineID, output.DestinationID)
|
||||
}
|
||||
if output.Source.ID != req.SourceBundle.Manifest.ID || output.Source.Digest != req.SourceBundle.Manifest.Digest || !output.Source.Created.Equal(req.SourceBundle.Manifest.Created) {
|
||||
t.Fatalf("catalog output source = %#v", output.Source)
|
||||
}
|
||||
if output.Kind == state.OutputKindSource && output.SourcePath != "" {
|
||||
t.Fatalf("source catalog output source_path = %q, want empty", output.SourcePath)
|
||||
}
|
||||
if !output.CreatedAt.Equal(planUpdatedAt) || !output.UpdatedAt.Equal(planUpdatedAt) {
|
||||
t.Fatalf("catalog output times = %s/%s", output.CreatedAt, output.UpdatedAt)
|
||||
}
|
||||
}
|
||||
if _, err := destinationBackend.Stat(context.Background(), storage.StateFileName); !storage.IsNotFound(err) {
|
||||
t.Fatalf("destination state stat error = %v, want missing", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildAdditiveOverwritesManagedAndRetainsOthers(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
existing := baseCatalog(req)
|
||||
existing.Outputs = []state.CatalogOutputFile{
|
||||
catalogOutput(req, "reports", "archive", "report.md", state.OutputKindSource, planCreatedAt),
|
||||
catalogOutput(req, "reports", "web", "old.txt", state.OutputKindSource, planCreatedAt),
|
||||
}
|
||||
writeCatalogState(t, destinationBackend, "", existing)
|
||||
testutil.WriteFakeFile(t, destinationBackend, "report.md", "old")
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionUpsertAdditive {
|
||||
t.Fatalf("plan action = %s, want %s", plan.Action, ActionUpsertAdditive)
|
||||
}
|
||||
if len(plan.CatalogOutputsToDelete) != 0 {
|
||||
t.Fatalf("delete outputs = %#v, want none", plan.CatalogOutputsToDelete)
|
||||
}
|
||||
if len(plan.CatalogOutputsToRetain) != 1 || plan.CatalogOutputsToRetain[0].Path != "old.txt" {
|
||||
t.Fatalf("retained outputs = %#v, want old.txt", plan.CatalogOutputsToRetain)
|
||||
}
|
||||
written, ok := state.FindCatalogOutputByPath(plan.CatalogOutputsToWrite, "report.md")
|
||||
if !ok {
|
||||
t.Fatalf("written outputs = %#v, want report.md", plan.CatalogOutputsToWrite)
|
||||
}
|
||||
if !written.CreatedAt.Equal(planCreatedAt) || !written.UpdatedAt.Equal(planUpdatedAt) {
|
||||
t.Fatalf("report.md times = %s/%s, want created preserved and updated now", written.CreatedAt, written.UpdatedAt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildReplacementDeletesCurrentOwnerAndRetainsOtherOwners(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowReplacement)
|
||||
existing := baseCatalog(req)
|
||||
existing.Outputs = []state.CatalogOutputFile{
|
||||
catalogOutput(req, "reports", "archive", "report.md", state.OutputKindSource, planCreatedAt),
|
||||
catalogOutput(req, "reports", "archive", "stale.txt", state.OutputKindSource, planCreatedAt),
|
||||
catalogOutput(req, "reports", "web", "shared.txt", state.OutputKindSource, planCreatedAt),
|
||||
}
|
||||
writeCatalogState(t, destinationBackend, "", existing)
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionReplaceCatalog {
|
||||
t.Fatalf("plan action = %s, want %s", plan.Action, ActionReplaceCatalog)
|
||||
}
|
||||
if len(plan.CatalogOutputsToDelete) != 1 || plan.CatalogOutputsToDelete[0].Path != "stale.txt" {
|
||||
t.Fatalf("delete outputs = %#v, want stale.txt", plan.CatalogOutputsToDelete)
|
||||
}
|
||||
if len(plan.CatalogOutputsToRetain) != 1 || plan.CatalogOutputsToRetain[0].Path != "shared.txt" {
|
||||
t.Fatalf("retained outputs = %#v, want shared.txt", plan.CatalogOutputsToRetain)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildTransfersManagedPathOwnership(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowReplacement)
|
||||
existing := baseCatalog(req)
|
||||
existing.Outputs = []state.CatalogOutputFile{
|
||||
catalogOutput(req, "reports", "web", "report.md", state.OutputKindSource, planCreatedAt),
|
||||
}
|
||||
writeCatalogState(t, destinationBackend, "", existing)
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
written, ok := state.FindCatalogOutputByPath(plan.CatalogOutputsToWrite, "report.md")
|
||||
if !ok {
|
||||
t.Fatalf("written outputs = %#v, want report.md", plan.CatalogOutputsToWrite)
|
||||
}
|
||||
if written.PipelineID != "reports" || written.DestinationID != "archive" {
|
||||
t.Fatalf("written owner = %s/%s, want reports/archive", written.PipelineID, written.DestinationID)
|
||||
}
|
||||
if !written.CreatedAt.Equal(planCreatedAt) || !written.UpdatedAt.Equal(planUpdatedAt) {
|
||||
t.Fatalf("written times = %s/%s", written.CreatedAt, written.UpdatedAt)
|
||||
}
|
||||
if len(plan.CatalogOutputsToRetain) != 0 || len(plan.CatalogOutputsToDelete) != 0 {
|
||||
t.Fatalf("retain=%#v delete=%#v, want no old record for overwritten path", plan.CatalogOutputsToRetain, plan.CatalogOutputsToDelete)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRejectsUnmanagedPlannedPathCollision(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
testutil.WriteFakeFile(t, destinationBackend, "report.md", "unmanaged")
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want unmanaged collision")
|
||||
}
|
||||
if plan.Action != ActionFailUnmanaged || !strings.Contains(err.Error(), "not managed by catalog state") {
|
||||
t.Fatalf("plan action=%s error=%v, want unmanaged catalog collision", plan.Action, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPlansSupersededLegacyAdditive(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
legacyState := testutil.DestinationState(req.SourceBundle.Manifest, testutil.DestinationStateOptions{})
|
||||
writeJSONState(t, destinationBackend, "", legacyState)
|
||||
testutil.WriteFakeFile(t, destinationBackend, "report.md", "legacy")
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.SupersededLegacy == nil || plan.SupersededLegacy.SchemaVersion != state.SchemaVersion {
|
||||
t.Fatalf("superseded legacy = %#v", plan.SupersededLegacy)
|
||||
}
|
||||
if plan.Action != ActionUpsertAdditive || plan.ClearDestinationRoot {
|
||||
t.Fatalf("plan action=%s clear=%t, want additive overwrite without clear", plan.Action, plan.ClearDestinationRoot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPlansSupersededLegacyReplacementClear(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowReplacement)
|
||||
legacyState := testutil.DestinationState(req.SourceBundle.Manifest, testutil.DestinationStateOptions{})
|
||||
writeJSONState(t, destinationBackend, "", legacyState)
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionReplaceCatalog || !plan.ClearDestinationRoot {
|
||||
t.Fatalf("plan action=%s clear=%t, want replacement clear", plan.Action, plan.ClearDestinationRoot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRejectsInvalidOrFutureState(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
data string
|
||||
}{
|
||||
{name: "invalid json", data: `{"schema_version":`},
|
||||
{name: "future schema", data: `{"schema_version":99}`},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, destinationBackend, req := catalogPlanRequest(t, config.WorkflowAdditive)
|
||||
testutil.WriteFakeFile(t, destinationBackend, storage.StateFileName, tt.data)
|
||||
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want conflict")
|
||||
}
|
||||
if plan.Action != ActionFailConflict {
|
||||
t.Fatalf("plan action = %s, want %s", plan.Action, ActionFailConflict)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRequestRejectsInvalidWorkflow(t *testing.T) {
|
||||
sourceBackend, destinationBackend, req := catalogPlanRequest(t, "append")
|
||||
req.SourceBackend = sourceBackend
|
||||
req.DestinationBackend = destinationBackend
|
||||
|
||||
err := validateRequest(req)
|
||||
if err == nil {
|
||||
t.Fatal("validateRequest() error = nil, want invalid workflow")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "destination.workflow") {
|
||||
t.Fatalf("validateRequest() error = %v, want workflow context", err)
|
||||
}
|
||||
}
|
||||
|
||||
func catalogPlanRequest(t *testing.T, workflow string) (*fake.Backend, *fake.Backend, Request) {
|
||||
t.Helper()
|
||||
sourceBackend := fake.New()
|
||||
destinationBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{})
|
||||
return sourceBackend, destinationBackend, Request{
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
SourceBundle: sourceBundle,
|
||||
SourceBackend: sourceBackend,
|
||||
DestinationBackend: destinationBackend,
|
||||
DestinationBundlePath: "",
|
||||
PathMapping: config.PathMappingPreserveRelative,
|
||||
Publish: config.PublishPolicy{Source: true},
|
||||
Workflow: workflow,
|
||||
DistributorVersion: "test",
|
||||
Now: planUpdatedAt,
|
||||
}
|
||||
}
|
||||
|
||||
func baseCatalog(req Request) state.CatalogState {
|
||||
return state.CatalogState{
|
||||
SchemaVersion: state.CatalogSchemaVersion,
|
||||
DistributorVersion: "previous",
|
||||
CreatedAt: planCreatedAt,
|
||||
UpdatedAt: planCreatedAt,
|
||||
State: state.StatePolicy{Mode: state.StateModeCatalog},
|
||||
Outputs: []state.CatalogOutputFile{},
|
||||
}
|
||||
}
|
||||
|
||||
func catalogOutput(req Request, pipelineID, destinationID, path, kind string, createdAt time.Time) state.CatalogOutputFile {
|
||||
sourceSHA := req.SourceBundle.Manifest.Files[0].SHA256
|
||||
sourceSize := req.SourceBundle.Manifest.Files[0].Size
|
||||
for _, file := range req.SourceBundle.Manifest.Files {
|
||||
if file.Path == path {
|
||||
sourceSHA = file.SHA256
|
||||
sourceSize = file.Size
|
||||
break
|
||||
}
|
||||
}
|
||||
output := state.CatalogOutputFile{
|
||||
Path: path,
|
||||
PipelineID: pipelineID,
|
||||
DestinationID: destinationID,
|
||||
Source: state.CatalogSourceIdentity{
|
||||
ID: req.SourceBundle.Manifest.ID,
|
||||
Digest: req.SourceBundle.Manifest.Digest,
|
||||
Created: req.SourceBundle.Manifest.Created,
|
||||
},
|
||||
Kind: kind,
|
||||
SHA256: sourceSHA,
|
||||
Size: sourceSize,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
}
|
||||
return output
|
||||
}
|
||||
|
||||
func writeCatalogState(t *testing.T, backend *fake.Backend, relative string, catalog state.CatalogState) {
|
||||
t.Helper()
|
||||
writeJSONState(t, backend, relative, catalog)
|
||||
}
|
||||
|
||||
func writeJSONState(t *testing.T, backend *fake.Backend, relative string, value any) {
|
||||
t.Helper()
|
||||
data, err := json.MarshalIndent(value, "", " ")
|
||||
if err != nil {
|
||||
t.Fatalf("marshal state: %v", err)
|
||||
}
|
||||
statePath, err := storage.StatePath(relative)
|
||||
if err != nil {
|
||||
t.Fatalf("state path: %v", err)
|
||||
}
|
||||
testutil.WriteFakeFile(t, backend, statePath, string(append(data, '\n')))
|
||||
}
|
||||
@@ -18,7 +18,13 @@ func inspectDestination(ctx context.Context, backend storage.Backend, bundlePath
|
||||
if parseErr != nil {
|
||||
return state.DestinationStatus{StateErr: parseErr}, nil
|
||||
}
|
||||
return state.DestinationStatus{State: document.SingleOwner, SharedRoot: document.SharedRoot, HasContents: true}, nil
|
||||
return state.DestinationStatus{
|
||||
State: document.SingleOwner,
|
||||
SharedRoot: document.SharedRoot,
|
||||
Catalog: document.Catalog,
|
||||
SupersededLegacy: document.SupersededLegacy,
|
||||
HasContents: true,
|
||||
}, nil
|
||||
}
|
||||
if !storage.IsNotFound(err) {
|
||||
return state.DestinationStatus{}, err
|
||||
|
||||
@@ -1,399 +0,0 @@
|
||||
package publish
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
|
||||
)
|
||||
|
||||
func TestBuildSharedRootTreatsAbsentOwnerAsPublishable(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", sharedRootStateWithOwners(t, sourceBundle.Manifest, false))
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionPublishNew {
|
||||
t.Fatalf("plan action = %s, want publish_new", plan.Action)
|
||||
}
|
||||
if plan.OwnerScope != state.CurrentOwnerScope("reports", "archive") {
|
||||
t.Fatalf("owner scope = %#v, want reports/archive", plan.OwnerScope)
|
||||
}
|
||||
if got, want := len(plan.OtherOwnerOutputs), 1; got != want {
|
||||
t.Fatalf("other owner output count = %d, want %d", got, want)
|
||||
}
|
||||
if got, want := len(plan.OwnerOutputsToWrite), 1; got != want {
|
||||
t.Fatalf("owner output write count = %d, want %d", got, want)
|
||||
}
|
||||
if len(plan.OwnerOutputsToDelete) != 0 || len(plan.RetainedOwnerOutputs) != 0 {
|
||||
t.Fatalf("delete=%#v retained=%#v, want none", plan.OwnerOutputsToDelete, plan.RetainedOwnerOutputs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildSharedRootReplaceDeletesOnlyCurrentOwnerOmittedOutputs(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", sharedRootStateWithOwners(t, sourceBundle.Manifest, true))
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionReplaceOlder {
|
||||
t.Fatalf("plan action = %s, want replace_older", plan.Action)
|
||||
}
|
||||
if got, want := sharedRootOutputPathList(plan.OwnerOutputsToDelete), "old.md"; got != want {
|
||||
t.Fatalf("owner outputs to delete = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := sharedRootOutputPathList(plan.OtherOwnerOutputs), "other/report.md"; got != want {
|
||||
t.Fatalf("other owner outputs = %q, want %q", got, want)
|
||||
}
|
||||
if len(plan.RetainedOwnerOutputs) != 0 {
|
||||
t.Fatalf("retained owner outputs = %#v, want none", plan.RetainedOwnerOutputs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildSharedRootMergeRetainsCurrentOwnerOmittedOutputs(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", sharedRootStateWithOwners(t, sourceBundle.Manifest, true))
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeMerge)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if got, want := sharedRootOutputPathList(plan.RetainedOwnerOutputs), "old.md"; got != want {
|
||||
t.Fatalf("retained owner outputs = %q, want %q", got, want)
|
||||
}
|
||||
if len(plan.OwnerOutputsToDelete) != 0 {
|
||||
t.Fatalf("owner outputs to delete = %#v, want none", plan.OwnerOutputsToDelete)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildSharedRootRejectsOtherOwnerPathConflict(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
sharedRoot := sharedRootStateWithOwners(t, sourceBundle.Manifest, false)
|
||||
sharedRoot.Outputs[0].Path = "report.md"
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", sharedRoot)
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err == nil || !strings.Contains(err.Error(), "fail_conflict") {
|
||||
t.Fatalf("Build() error = %v, want fail_conflict", err)
|
||||
}
|
||||
if plan.Action != ActionFailConflict {
|
||||
t.Fatalf("plan action = %s, want fail_conflict", plan.Action)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildSharedRootRejectsUnmanagedPathCollision(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", sharedRootStateWithOwners(t, sourceBundle.Manifest, false))
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle/report.md", "unmanaged")
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err == nil || !strings.Contains(err.Error(), "fail_unmanaged") {
|
||||
t.Fatalf("Build() error = %v, want fail_unmanaged", err)
|
||||
}
|
||||
if plan.Action != ActionFailUnmanaged {
|
||||
t.Fatalf("plan action = %s, want fail_unmanaged", plan.Action)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteSharedRootPublishesOwnerAndPreservesOtherOwners(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", sharedRootStateWithOwners(t, sourceBundle.Manifest, false))
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/other/report.md", "old")
|
||||
destinationState := readFakeSharedRootState(t, destinationBackend, "bundle")
|
||||
if got, want := len(destinationState.Owners), 2; got != want {
|
||||
t.Fatalf("owner count = %d, want %d", got, want)
|
||||
}
|
||||
if got, want := destinationState.State.Mode, state.StateModeSharedRoot; got != want {
|
||||
t.Fatalf("state mode = %q, want %q", got, want)
|
||||
}
|
||||
if _, ok := destinationState.Owner(state.CurrentOwnerScope("other", "archive")); !ok {
|
||||
t.Fatal("other owner missing from shared-root state")
|
||||
}
|
||||
if _, ok := destinationState.Owner(state.CurrentOwnerScope("reports", "archive")); !ok {
|
||||
t.Fatal("current owner missing from shared-root state")
|
||||
}
|
||||
if got, want := strings.Join(destinationState.AllManagedOutputPaths(), ","), "other/report.md,report.md"; got != want {
|
||||
t.Fatalf("managed paths = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteSharedRootReplaceDeletesOnlyCurrentOwnerOmittedOutputs(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
existing := sharedRootStateWithOwners(t, sourceBundle.Manifest, true)
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", existing)
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\nNew.\n")
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "bundle/old.md")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/other/report.md", "old")
|
||||
destinationState := readFakeSharedRootState(t, destinationBackend, "bundle")
|
||||
if got, want := strings.Join(destinationState.AllManagedOutputPaths(), ","), "other/report.md,report.md"; got != want {
|
||||
t.Fatalf("managed paths = %q, want %q", got, want)
|
||||
}
|
||||
if !destinationState.CreatedAt.Equal(existing.CreatedAt) {
|
||||
t.Fatalf("created_at = %s, want %s", destinationState.CreatedAt, existing.CreatedAt)
|
||||
}
|
||||
output, ok := findSharedRootOutputForTest(destinationState.Outputs, "report.md")
|
||||
if !ok {
|
||||
t.Fatal("report.md missing from shared-root outputs")
|
||||
}
|
||||
if !output.CreatedAt.Equal(existing.Outputs[1].CreatedAt) || !output.UpdatedAt.After(existing.Outputs[1].UpdatedAt) {
|
||||
t.Fatalf("report.md timestamps = created:%s updated:%s, want preserved created and newer updated", output.CreatedAt, output.UpdatedAt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteSharedRootMergeRetainsCurrentOwnerOmittedOutputs(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\nNew.\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
writeFakeSharedRootState(t, destinationBackend, "bundle", sharedRootStateWithOwners(t, sourceBundle.Manifest, true))
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeMerge)
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\nNew.\n")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/old.md", "old")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/other/report.md", "old")
|
||||
destinationState := readFakeSharedRootState(t, destinationBackend, "bundle")
|
||||
if got, want := strings.Join(destinationState.AllManagedOutputPaths(), ","), "other/report.md,report.md,old.md"; got != want {
|
||||
t.Fatalf("managed paths = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteSharedRootForceReplaceDeletesOnlyBundlePath(t *testing.T) {
|
||||
sourceBackend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{
|
||||
Files: []testutil.SourceFile{{Path: "report.md", Data: "# Report\n"}},
|
||||
})
|
||||
destinationBackend := fake.New()
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle/unmanaged.txt", "unmanaged")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle/nested/old.txt", "old")
|
||||
testutil.WriteFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
|
||||
|
||||
req := sharedRootRequest(sourceBackend, destinationBackend, sourceBundle, config.ReconciliationModeReplace)
|
||||
req.Force = true
|
||||
plan, err := Build(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if plan.Action != ActionForceReplace {
|
||||
t.Fatalf("plan action = %s, want force_replace", plan.Action)
|
||||
}
|
||||
if err := Execute(context.Background(), req, plan); err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\n")
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "bundle/unmanaged.txt")
|
||||
testutil.AssertFakeMissing(t, destinationBackend, "bundle/nested/old.txt")
|
||||
testutil.AssertFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
|
||||
destinationState := readFakeSharedRootState(t, destinationBackend, "bundle")
|
||||
if got, want := len(destinationState.Owners), 1; got != want {
|
||||
t.Fatalf("owner count = %d, want %d", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func sharedRootRequest(sourceBackend, destinationBackend *fake.Backend, sourceBundle bundle.Bundle, reconciliationMode string) Request {
|
||||
return Request{
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
SourceBundle: sourceBundle,
|
||||
SourceBackend: sourceBackend,
|
||||
DestinationBackend: destinationBackend,
|
||||
DestinationBundlePath: sourceBundle.RootRelativePath,
|
||||
Publish: config.PublishPolicy{Source: true},
|
||||
State: config.StatePolicy{Mode: config.StateModeSharedRoot},
|
||||
Reconciliation: config.ReconciliationPolicy{Mode: reconciliationMode},
|
||||
Transfer: defaultTransfer(),
|
||||
DistributorVersion: "test",
|
||||
}
|
||||
}
|
||||
|
||||
func writeFakeSharedRootState(t *testing.T, backend *fake.Backend, relative string, sharedRoot state.SharedRootState) {
|
||||
t.Helper()
|
||||
data, err := json.MarshalIndent(sharedRoot, "", " ")
|
||||
if err != nil {
|
||||
t.Fatalf("marshal shared-root state: %v", err)
|
||||
}
|
||||
statePath, err := storage.StatePath(relative)
|
||||
if err != nil {
|
||||
t.Fatalf("state path: %v", err)
|
||||
}
|
||||
testutil.WriteFakeFile(t, backend, statePath, string(append(data, '\n')))
|
||||
for _, output := range sharedRoot.Outputs {
|
||||
path, err := storage.Join(relative, output.Path)
|
||||
if err != nil {
|
||||
t.Fatalf("join output path: %v", err)
|
||||
}
|
||||
testutil.WriteFakeFile(t, backend, path, "old")
|
||||
}
|
||||
}
|
||||
|
||||
func readFakeSharedRootState(t *testing.T, backend *fake.Backend, relative string) state.SharedRootState {
|
||||
t.Helper()
|
||||
statePath, err := storage.StatePath(relative)
|
||||
if err != nil {
|
||||
t.Fatalf("state path: %v", err)
|
||||
}
|
||||
data, err := backend.ReadFile(context.Background(), statePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read shared-root state: %v", err)
|
||||
}
|
||||
destinationState, err := state.ParseSharedRoot(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse shared-root state: %v", err)
|
||||
}
|
||||
return destinationState
|
||||
}
|
||||
|
||||
func findSharedRootOutputForTest(outputs []state.SharedRootOutputFile, path string) (state.SharedRootOutputFile, bool) {
|
||||
for _, output := range outputs {
|
||||
if output.Path == path {
|
||||
return output, true
|
||||
}
|
||||
}
|
||||
return state.SharedRootOutputFile{}, false
|
||||
}
|
||||
|
||||
func sharedRootStateWithOwners(t *testing.T, current bundle.Manifest, includeCurrent bool) state.SharedRootState {
|
||||
t.Helper()
|
||||
createdAt := time.Date(2026, 5, 30, 11, 12, 0, 0, time.UTC)
|
||||
otherManifest := testutil.ValidManifest(testutil.BundleOptions{
|
||||
ID: "other.source",
|
||||
Files: []testutil.SourceFile{{Path: "other/report.md", Data: "# Other\n"}},
|
||||
})
|
||||
sharedRoot := state.SharedRootState{
|
||||
SchemaVersion: state.SharedRootSchemaVersion,
|
||||
DistributorVersion: "test",
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
State: state.StatePolicy{Mode: state.StateModeSharedRoot},
|
||||
Owners: []state.OwnerRecord{{
|
||||
Scope: state.CurrentOwnerScope("other", "archive"),
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: config.ReconciliationModeReplace},
|
||||
Source: state.SourceState{Manifest: otherManifest},
|
||||
}},
|
||||
Outputs: []state.SharedRootOutputFile{{
|
||||
Path: "other/report.md",
|
||||
Kind: state.OutputKindSource,
|
||||
SourcePath: "other/report.md",
|
||||
SHA256: otherManifest.Files[0].SHA256,
|
||||
Size: otherManifest.Files[0].Size,
|
||||
Owner: state.CurrentOwnerScope("other", "archive"),
|
||||
SourceID: otherManifest.ID,
|
||||
SourceDigest: otherManifest.Digest,
|
||||
SourceCreated: otherManifest.Created,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
}},
|
||||
}
|
||||
if !includeCurrent {
|
||||
return sharedRoot
|
||||
}
|
||||
older := current
|
||||
older.Created = older.Created.Add(-time.Hour)
|
||||
older.Files = append([]bundle.ManifestFile(nil), current.Files...)
|
||||
older.Files = append(older.Files, bundle.ManifestFile{
|
||||
Path: "old.md",
|
||||
SHA256: bundle.FileDigest([]byte("old\n")),
|
||||
Size: int64(len("old\n")),
|
||||
})
|
||||
older.Digest = bundle.BundleDigest(older.Files)
|
||||
scope := state.CurrentOwnerScope("reports", "archive")
|
||||
sharedRoot.Owners = append(sharedRoot.Owners, state.OwnerRecord{
|
||||
Scope: scope,
|
||||
Reconciliation: state.ReconciliationPolicy{Mode: config.ReconciliationModeReplace},
|
||||
Source: state.SourceState{Manifest: older},
|
||||
})
|
||||
for _, file := range older.Files {
|
||||
sharedRoot.Outputs = append(sharedRoot.Outputs, state.SharedRootOutputFile{
|
||||
Path: file.Path,
|
||||
Kind: state.OutputKindSource,
|
||||
SourcePath: file.Path,
|
||||
SHA256: file.SHA256,
|
||||
Size: file.Size,
|
||||
Owner: scope,
|
||||
SourceID: older.ID,
|
||||
SourceDigest: older.Digest,
|
||||
SourceCreated: older.Created,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: createdAt,
|
||||
})
|
||||
}
|
||||
return sharedRoot
|
||||
}
|
||||
|
||||
func sharedRootOutputPathList(outputs []state.SharedRootOutputFile) string {
|
||||
paths := make([]string, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
paths = append(paths, output.Path)
|
||||
}
|
||||
return strings.Join(paths, ",")
|
||||
}
|
||||
422
internal/state/catalog.go
Normal file
422
internal/state/catalog.go
Normal file
@@ -0,0 +1,422 @@
|
||||
package state
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/link"
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||
)
|
||||
|
||||
type CatalogState struct {
|
||||
SchemaVersion int
|
||||
DistributorVersion string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
State StatePolicy
|
||||
Outputs []CatalogOutputFile
|
||||
}
|
||||
|
||||
type CatalogSourceIdentity struct {
|
||||
ID string
|
||||
Digest string
|
||||
Created time.Time
|
||||
}
|
||||
|
||||
type CatalogOutputFile struct {
|
||||
Path string
|
||||
PipelineID string
|
||||
DestinationID string
|
||||
Source CatalogSourceIdentity
|
||||
Kind string
|
||||
SourcePath string
|
||||
Transform string
|
||||
URL string
|
||||
SHA256 string
|
||||
Size int64
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
type rawCatalogState struct {
|
||||
SchemaVersion *int `json:"schema_version"`
|
||||
DistributorVersion string `json:"distributor_version"`
|
||||
CreatedAt *string `json:"created_at"`
|
||||
UpdatedAt *string `json:"updated_at"`
|
||||
State *rawStatePolicy `json:"state"`
|
||||
Outputs []rawCatalogOutput `json:"outputs"`
|
||||
}
|
||||
|
||||
type rawCatalogOutput struct {
|
||||
Path *string `json:"path"`
|
||||
PipelineID *string `json:"pipeline_id"`
|
||||
DestinationID *string `json:"destination_id"`
|
||||
Source *rawCatalogSourceIdentity `json:"source"`
|
||||
Kind *string `json:"kind"`
|
||||
SourcePath *string `json:"source_path"`
|
||||
Transform *string `json:"transform"`
|
||||
URL *string `json:"url"`
|
||||
SHA256 *string `json:"sha256"`
|
||||
Size *int64 `json:"size"`
|
||||
CreatedAt *string `json:"created_at"`
|
||||
UpdatedAt *string `json:"updated_at"`
|
||||
}
|
||||
|
||||
type rawCatalogSourceIdentity struct {
|
||||
ID *string `json:"id"`
|
||||
Digest *string `json:"digest"`
|
||||
Created *string `json:"created"`
|
||||
}
|
||||
|
||||
func ParseCatalog(data []byte) (CatalogState, error) {
|
||||
decoder := json.NewDecoder(bytes.NewReader(data))
|
||||
decoder.DisallowUnknownFields()
|
||||
var raw rawCatalogState
|
||||
if err := decoder.Decode(&raw); err != nil {
|
||||
return CatalogState{}, fmt.Errorf("parse distributor state: %w", err)
|
||||
}
|
||||
var extra any
|
||||
if err := decoder.Decode(&extra); err != io.EOF {
|
||||
return CatalogState{}, fmt.Errorf("parse distributor state: trailing data")
|
||||
}
|
||||
state, err := parseCatalogRaw(raw)
|
||||
if err != nil {
|
||||
return CatalogState{}, err
|
||||
}
|
||||
if err := ValidateCatalog(state); err != nil {
|
||||
return CatalogState{}, err
|
||||
}
|
||||
return state, nil
|
||||
}
|
||||
|
||||
func parseCatalogRaw(raw rawCatalogState) (CatalogState, error) {
|
||||
if raw.SchemaVersion == nil {
|
||||
return CatalogState{}, fmt.Errorf("state schema_version is required")
|
||||
}
|
||||
state := CatalogState{
|
||||
SchemaVersion: *raw.SchemaVersion,
|
||||
DistributorVersion: raw.DistributorVersion,
|
||||
}
|
||||
if state.SchemaVersion != CatalogSchemaVersion {
|
||||
return CatalogState{}, fmt.Errorf("state schema_version must be %d", CatalogSchemaVersion)
|
||||
}
|
||||
createdAt, err := parseRequiredTime("state created_at", raw.CreatedAt)
|
||||
if err != nil {
|
||||
return CatalogState{}, err
|
||||
}
|
||||
updatedAt, err := parseRequiredTime("state updated_at", raw.UpdatedAt)
|
||||
if err != nil {
|
||||
return CatalogState{}, err
|
||||
}
|
||||
state.CreatedAt = createdAt
|
||||
state.UpdatedAt = updatedAt
|
||||
if raw.State == nil || raw.State.Mode == "" {
|
||||
return CatalogState{}, fmt.Errorf("state state.mode is required")
|
||||
}
|
||||
state.State.Mode = raw.State.Mode
|
||||
if raw.Outputs == nil {
|
||||
return CatalogState{}, fmt.Errorf("state outputs is required")
|
||||
}
|
||||
outputs, err := parseCatalogOutputs(raw.Outputs)
|
||||
if err != nil {
|
||||
return CatalogState{}, err
|
||||
}
|
||||
state.Outputs = outputs
|
||||
return state, nil
|
||||
}
|
||||
|
||||
func parseCatalogOutputs(rawOutputs []rawCatalogOutput) ([]CatalogOutputFile, error) {
|
||||
outputs := make([]CatalogOutputFile, 0, len(rawOutputs))
|
||||
for index, raw := range rawOutputs {
|
||||
output, err := parseCatalogOutput(index, raw)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
outputs = append(outputs, output)
|
||||
}
|
||||
return outputs, nil
|
||||
}
|
||||
|
||||
func parseCatalogOutput(index int, raw rawCatalogOutput) (CatalogOutputFile, error) {
|
||||
if raw.Path == nil || *raw.Path == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].path is required", index)
|
||||
}
|
||||
if raw.PipelineID == nil || *raw.PipelineID == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].pipeline_id is required", index)
|
||||
}
|
||||
if raw.DestinationID == nil || *raw.DestinationID == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].destination_id is required", index)
|
||||
}
|
||||
source, err := parseCatalogSourceIdentity(index, raw.Source)
|
||||
if err != nil {
|
||||
return CatalogOutputFile{}, err
|
||||
}
|
||||
if raw.Kind == nil || *raw.Kind == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].kind is required", index)
|
||||
}
|
||||
switch *raw.Kind {
|
||||
case OutputKindSource:
|
||||
if raw.SourcePath != nil {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].source_path is only valid for generated output", index)
|
||||
}
|
||||
if raw.Transform != nil {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].transform is only valid for generated output", index)
|
||||
}
|
||||
case OutputKindGenerated:
|
||||
if raw.SourcePath == nil || *raw.SourcePath == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].source_path is required for generated output", index)
|
||||
}
|
||||
if raw.Transform == nil || *raw.Transform == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].transform is required for generated output", index)
|
||||
}
|
||||
}
|
||||
if raw.SHA256 == nil || *raw.SHA256 == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].sha256 is required", index)
|
||||
}
|
||||
if raw.Size == nil {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].size is required", index)
|
||||
}
|
||||
createdAt, err := parseRequiredTime(fmt.Sprintf("state outputs[%d].created_at", index), raw.CreatedAt)
|
||||
if err != nil {
|
||||
return CatalogOutputFile{}, err
|
||||
}
|
||||
updatedAt, err := parseRequiredTime(fmt.Sprintf("state outputs[%d].updated_at", index), raw.UpdatedAt)
|
||||
if err != nil {
|
||||
return CatalogOutputFile{}, err
|
||||
}
|
||||
output := CatalogOutputFile{
|
||||
Path: *raw.Path,
|
||||
PipelineID: *raw.PipelineID,
|
||||
DestinationID: *raw.DestinationID,
|
||||
Source: source,
|
||||
Kind: *raw.Kind,
|
||||
SHA256: *raw.SHA256,
|
||||
Size: *raw.Size,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: updatedAt,
|
||||
}
|
||||
if raw.SourcePath != nil {
|
||||
output.SourcePath = *raw.SourcePath
|
||||
}
|
||||
if raw.Transform != nil {
|
||||
output.Transform = *raw.Transform
|
||||
}
|
||||
if raw.URL != nil {
|
||||
if *raw.URL == "" {
|
||||
return CatalogOutputFile{}, fmt.Errorf("state outputs[%d].url must not be empty", index)
|
||||
}
|
||||
output.URL = *raw.URL
|
||||
}
|
||||
return output, nil
|
||||
}
|
||||
|
||||
func parseCatalogSourceIdentity(index int, raw *rawCatalogSourceIdentity) (CatalogSourceIdentity, error) {
|
||||
if raw == nil {
|
||||
return CatalogSourceIdentity{}, fmt.Errorf("state outputs[%d].source is required", index)
|
||||
}
|
||||
if raw.ID == nil || *raw.ID == "" {
|
||||
return CatalogSourceIdentity{}, fmt.Errorf("state outputs[%d].source.id is required", index)
|
||||
}
|
||||
if raw.Digest == nil || *raw.Digest == "" {
|
||||
return CatalogSourceIdentity{}, fmt.Errorf("state outputs[%d].source.digest is required", index)
|
||||
}
|
||||
created, err := parseRequiredTime(fmt.Sprintf("state outputs[%d].source.created", index), raw.Created)
|
||||
if err != nil {
|
||||
return CatalogSourceIdentity{}, err
|
||||
}
|
||||
return CatalogSourceIdentity{
|
||||
ID: *raw.ID,
|
||||
Digest: *raw.Digest,
|
||||
Created: created,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s CatalogState) CreatedAtString() string {
|
||||
return s.CreatedAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func (s CatalogState) UpdatedAtString() string {
|
||||
return s.UpdatedAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func (s CatalogSourceIdentity) CreatedString() string {
|
||||
return s.Created.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func (o CatalogOutputFile) CreatedAtString() string {
|
||||
return o.CreatedAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func (o CatalogOutputFile) UpdatedAtString() string {
|
||||
return o.UpdatedAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func (s CatalogState) MarshalJSON() ([]byte, error) {
|
||||
type stateJSON struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
DistributorVersion string `json:"distributor_version,omitempty"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
State StatePolicy `json:"state"`
|
||||
Outputs []CatalogOutputFile `json:"outputs"`
|
||||
}
|
||||
return json.Marshal(stateJSON{
|
||||
SchemaVersion: s.SchemaVersion,
|
||||
DistributorVersion: s.DistributorVersion,
|
||||
CreatedAt: s.CreatedAtString(),
|
||||
UpdatedAt: s.UpdatedAtString(),
|
||||
State: s.State,
|
||||
Outputs: s.Outputs,
|
||||
})
|
||||
}
|
||||
|
||||
func (s CatalogSourceIdentity) MarshalJSON() ([]byte, error) {
|
||||
type sourceJSON struct {
|
||||
ID string `json:"id"`
|
||||
Digest string `json:"digest"`
|
||||
Created string `json:"created"`
|
||||
}
|
||||
return json.Marshal(sourceJSON{
|
||||
ID: s.ID,
|
||||
Digest: s.Digest,
|
||||
Created: s.CreatedString(),
|
||||
})
|
||||
}
|
||||
|
||||
func (o CatalogOutputFile) MarshalJSON() ([]byte, error) {
|
||||
type outputJSON struct {
|
||||
Path string `json:"path"`
|
||||
PipelineID string `json:"pipeline_id"`
|
||||
DestinationID string `json:"destination_id"`
|
||||
Source CatalogSourceIdentity `json:"source"`
|
||||
Kind string `json:"kind"`
|
||||
SourcePath string `json:"source_path,omitempty"`
|
||||
Transform string `json:"transform,omitempty"`
|
||||
URL string `json:"url,omitempty"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Size int64 `json:"size"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
return json.Marshal(outputJSON{
|
||||
Path: o.Path,
|
||||
PipelineID: o.PipelineID,
|
||||
DestinationID: o.DestinationID,
|
||||
Source: o.Source,
|
||||
Kind: o.Kind,
|
||||
SourcePath: o.SourcePath,
|
||||
Transform: o.Transform,
|
||||
URL: o.URL,
|
||||
SHA256: o.SHA256,
|
||||
Size: o.Size,
|
||||
CreatedAt: o.CreatedAtString(),
|
||||
UpdatedAt: o.UpdatedAtString(),
|
||||
})
|
||||
}
|
||||
|
||||
func ValidateCatalog(s CatalogState) error {
|
||||
if s.SchemaVersion != CatalogSchemaVersion {
|
||||
return fmt.Errorf("state schema_version must be %d", CatalogSchemaVersion)
|
||||
}
|
||||
if s.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("state created_at is required")
|
||||
}
|
||||
if s.UpdatedAt.IsZero() {
|
||||
return fmt.Errorf("state updated_at is required")
|
||||
}
|
||||
if s.State.Mode != StateModeCatalog {
|
||||
return fmt.Errorf("state state.mode must be %s", StateModeCatalog)
|
||||
}
|
||||
if s.Outputs == nil {
|
||||
return fmt.Errorf("state outputs is required")
|
||||
}
|
||||
seenPaths := make(map[string]struct{}, len(s.Outputs))
|
||||
for index, output := range s.Outputs {
|
||||
if err := validateCatalogOutput(index, output); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, exists := seenPaths[output.Path]; exists {
|
||||
return fmt.Errorf("state outputs[%d].path duplicates %q", index, output.Path)
|
||||
}
|
||||
seenPaths[output.Path] = struct{}{}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateCatalogOutput(index int, output CatalogOutputFile) error {
|
||||
if err := storage.ValidatePath(output.Path); err != nil {
|
||||
return fmt.Errorf("state outputs[%d].path: %w", index, err)
|
||||
}
|
||||
if output.PipelineID == "" {
|
||||
return fmt.Errorf("state outputs[%d].pipeline_id is required", index)
|
||||
}
|
||||
if !config.IsSlugLikeID(output.PipelineID) {
|
||||
return fmt.Errorf("state outputs[%d].pipeline_id must be a slug-like identifier", index)
|
||||
}
|
||||
if output.DestinationID == "" {
|
||||
return fmt.Errorf("state outputs[%d].destination_id is required", index)
|
||||
}
|
||||
if !config.IsSlugLikeID(output.DestinationID) {
|
||||
return fmt.Errorf("state outputs[%d].destination_id must be a slug-like identifier", index)
|
||||
}
|
||||
if err := validateCatalogSourceIdentity(index, output.Source); err != nil {
|
||||
return err
|
||||
}
|
||||
switch output.Kind {
|
||||
case OutputKindSource:
|
||||
if output.SourcePath != "" {
|
||||
return fmt.Errorf("state outputs[%d].source_path is only valid for generated output", index)
|
||||
}
|
||||
if output.Transform != "" {
|
||||
return fmt.Errorf("state outputs[%d].transform is only valid for generated output", index)
|
||||
}
|
||||
case OutputKindGenerated:
|
||||
if output.SourcePath == "" {
|
||||
return fmt.Errorf("state outputs[%d].source_path is required for generated output", index)
|
||||
}
|
||||
if err := storage.ValidatePath(output.SourcePath); err != nil {
|
||||
return fmt.Errorf("state outputs[%d].source_path: %w", index, err)
|
||||
}
|
||||
if output.Transform == "" {
|
||||
return fmt.Errorf("state outputs[%d].transform is required for generated output", index)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("state outputs[%d].kind must be source or generated", index)
|
||||
}
|
||||
if output.URL != "" {
|
||||
if err := link.ValidateHTTPURL(output.URL); err != nil {
|
||||
return fmt.Errorf("state outputs[%d].url: %w", index, err)
|
||||
}
|
||||
}
|
||||
if err := bundle.ValidateDigest(output.SHA256); err != nil {
|
||||
return fmt.Errorf("state outputs[%d].sha256: %w", index, err)
|
||||
}
|
||||
if output.Size < 0 {
|
||||
return fmt.Errorf("state outputs[%d].size must be non-negative", index)
|
||||
}
|
||||
if output.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("state outputs[%d].created_at is required", index)
|
||||
}
|
||||
if output.UpdatedAt.IsZero() {
|
||||
return fmt.Errorf("state outputs[%d].updated_at is required", index)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateCatalogSourceIdentity(index int, source CatalogSourceIdentity) error {
|
||||
if source.ID == "" {
|
||||
return fmt.Errorf("state outputs[%d].source.id is required", index)
|
||||
}
|
||||
if err := bundle.ValidateDigest(source.Digest); err != nil {
|
||||
return fmt.Errorf("state outputs[%d].source.digest: %w", index, err)
|
||||
}
|
||||
if source.Created.IsZero() {
|
||||
return fmt.Errorf("state outputs[%d].source.created is required", index)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
361
internal/state/catalog_test.go
Normal file
361
internal/state/catalog_test.go
Normal file
@@ -0,0 +1,361 @@
|
||||
package state
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestParseCatalogState(t *testing.T) {
|
||||
state, err := ParseCatalog([]byte(validCatalogStateJSON(t)))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseCatalog() error = %v", err)
|
||||
}
|
||||
if got, want := state.SchemaVersion, CatalogSchemaVersion; got != want {
|
||||
t.Fatalf("schema version = %d, want %d", got, want)
|
||||
}
|
||||
if got, want := state.CreatedAtString(), "2026-06-19T12:00:00Z"; got != want {
|
||||
t.Fatalf("created_at = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := state.UpdatedAtString(), "2026-06-19T12:05:00Z"; got != want {
|
||||
t.Fatalf("updated_at = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := state.State.Mode, StateModeCatalog; got != want {
|
||||
t.Fatalf("state mode = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := len(state.Outputs), 2; got != want {
|
||||
t.Fatalf("output count = %d, want %d", got, want)
|
||||
}
|
||||
sourceOutput := state.Outputs[0]
|
||||
if sourceOutput.SourcePath != "" || sourceOutput.Transform != "" {
|
||||
t.Fatalf("source output source_path=%q transform=%q, want omitted", sourceOutput.SourcePath, sourceOutput.Transform)
|
||||
}
|
||||
generatedOutput := state.Outputs[1]
|
||||
if generatedOutput.SourcePath != "report.md" || generatedOutput.Transform != "markdown_to_html" {
|
||||
t.Fatalf("generated output source_path=%q transform=%q", generatedOutput.SourcePath, generatedOutput.Transform)
|
||||
}
|
||||
if got, want := generatedOutput.Source.CreatedString(), "2026-05-30T11:10:00Z"; got != want {
|
||||
t.Fatalf("source created = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCatalogMarshalIsDeterministic(t *testing.T) {
|
||||
data, err := json.Marshal(validCatalogState(t))
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal() error = %v", err)
|
||||
}
|
||||
want := `{"schema_version":4,"distributor_version":"dev","created_at":"2026-06-19T12:00:00Z","updated_at":"2026-06-19T12:05:00Z","state":{"mode":"catalog"},"outputs":[{"path":"report.md","pipeline_id":"reports","destination_id":"archive","source":{"id":"weather.daily.brentwood.2026-05-30","digest":"sha256:099b205780d2b050024868399961b05731729a548d5d6329c7b06a6740dd75fe","created":"2026-05-30T11:10:00Z"},"kind":"source","sha256":"sha256:3640fd37140ee4d2e0e93e78834f232ea67a50e7bc6279203690cc7de1975fa6","size":16,"created_at":"2026-06-19T12:00:00Z","updated_at":"2026-06-19T12:05:00Z"},{"path":"report.html","pipeline_id":"reports","destination_id":"html","source":{"id":"weather.daily.brentwood.2026-05-30","digest":"sha256:099b205780d2b050024868399961b05731729a548d5d6329c7b06a6740dd75fe","created":"2026-05-30T11:10:00Z"},"kind":"generated","source_path":"report.md","transform":"markdown_to_html","url":"https://reports.example.com/report.html","sha256":"sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","size":128,"created_at":"2026-06-19T12:00:00Z","updated_at":"2026-06-19T12:05:00Z"}]}`
|
||||
if string(data) != want {
|
||||
t.Fatalf("json = %s, want %s", data, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDocumentHandlesCatalogAndSupersededLegacy(t *testing.T) {
|
||||
catalog, err := ParseDocument([]byte(validCatalogStateJSON(t)))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseDocument(catalog) error = %v", err)
|
||||
}
|
||||
if catalog.Catalog == nil || catalog.SingleOwner != nil || catalog.SharedRoot != nil || catalog.SupersededLegacy != nil {
|
||||
t.Fatalf("catalog document = %#v", catalog)
|
||||
}
|
||||
|
||||
tests := map[string]string{
|
||||
"schema 1": legacyStateJSON(t),
|
||||
"schema 2": validStateJSON(t),
|
||||
"schema 3": validSharedRootStateJSON(t),
|
||||
}
|
||||
for name, body := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
document, err := ParseDocument([]byte(body))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseDocument() error = %v", err)
|
||||
}
|
||||
if document.SupersededLegacy == nil || document.Catalog != nil || document.SingleOwner != nil || document.SharedRoot != nil {
|
||||
t.Fatalf("document = %#v, want superseded legacy only", document)
|
||||
}
|
||||
if document.SupersededLegacy.SchemaVersion < legacySchemaVersion || document.SupersededLegacy.SchemaVersion >= CatalogSchemaVersion {
|
||||
t.Fatalf("legacy schema version = %d, want 1 through 3", document.SupersededLegacy.SchemaVersion)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDocumentRejectsUnsupportedFutureSchema(t *testing.T) {
|
||||
body := strings.Replace(validCatalogStateJSON(t), `"schema_version": 4`, `"schema_version": 5`, 1)
|
||||
_, err := ParseDocument([]byte(body))
|
||||
assertStateErrorContains(t, err, "schema_version 5 is unsupported")
|
||||
}
|
||||
|
||||
func TestParseDocumentRejectsTrailingData(t *testing.T) {
|
||||
_, err := ParseDocument([]byte(validCatalogStateJSON(t) + "\n{}"))
|
||||
assertStateErrorContains(t, err, "trailing data")
|
||||
}
|
||||
|
||||
func TestParseCatalogRejectsMissingFields(t *testing.T) {
|
||||
tests := map[string]func(map[string]any){
|
||||
"schema_version": func(document map[string]any) {
|
||||
delete(document, "schema_version")
|
||||
},
|
||||
"created_at": func(document map[string]any) {
|
||||
delete(document, "created_at")
|
||||
},
|
||||
"updated_at": func(document map[string]any) {
|
||||
delete(document, "updated_at")
|
||||
},
|
||||
"state": func(document map[string]any) {
|
||||
delete(document, "state")
|
||||
},
|
||||
"outputs": func(document map[string]any) {
|
||||
delete(document, "outputs")
|
||||
},
|
||||
"path": func(document map[string]any) {
|
||||
delete(firstCatalogOutput(document), "path")
|
||||
},
|
||||
"pipeline_id": func(document map[string]any) {
|
||||
delete(firstCatalogOutput(document), "pipeline_id")
|
||||
},
|
||||
"destination_id": func(document map[string]any) {
|
||||
delete(firstCatalogOutput(document), "destination_id")
|
||||
},
|
||||
"source": func(document map[string]any) {
|
||||
delete(firstCatalogOutput(document), "source")
|
||||
},
|
||||
"source id": func(document map[string]any) {
|
||||
delete(firstCatalogSource(document), "id")
|
||||
},
|
||||
"source digest": func(document map[string]any) {
|
||||
delete(firstCatalogSource(document), "digest")
|
||||
},
|
||||
"source created": func(document map[string]any) {
|
||||
delete(firstCatalogSource(document), "created")
|
||||
},
|
||||
"kind": func(document map[string]any) {
|
||||
delete(firstCatalogOutput(document), "kind")
|
||||
},
|
||||
"sha256": func(document map[string]any) {
|
||||
delete(firstCatalogOutput(document), "sha256")
|
||||
},
|
||||
"size": func(document map[string]any) {
|
||||
delete(firstCatalogOutput(document), "size")
|
||||
},
|
||||
}
|
||||
for name, mutate := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
document := catalogStateObject(t)
|
||||
mutate(document)
|
||||
_, err := ParseCatalog(mustMarshalCatalogObject(t, document))
|
||||
assertStateErrorContains(t, err, "required")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCatalogRejectsMalformedTimestamps(t *testing.T) {
|
||||
tests := map[string]func(string) string{
|
||||
"created_at": func(body string) string {
|
||||
return strings.Replace(body, `"created_at": "2026-06-19T12:00:00Z"`, `"created_at": "June 19"`, 1)
|
||||
},
|
||||
"updated_at": func(body string) string {
|
||||
return strings.Replace(body, `"updated_at": "2026-06-19T12:05:00Z"`, `"updated_at": "June 19"`, 1)
|
||||
},
|
||||
"source created": func(body string) string {
|
||||
return strings.Replace(body, `"created": "2026-05-30T11:10:00Z"`, `"created": "May 30"`, 1)
|
||||
},
|
||||
"output created_at": func(body string) string {
|
||||
return strings.Replace(body, ` "created_at": "2026-06-19T12:00:00Z"`, ` "created_at": "June 19"`, 1)
|
||||
},
|
||||
"output updated_at": func(body string) string {
|
||||
return strings.Replace(body, ` "updated_at": "2026-06-19T12:05:00Z"`, ` "updated_at": "June 19"`, 1)
|
||||
},
|
||||
}
|
||||
for name, mutate := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
_, err := ParseCatalog([]byte(mutate(validCatalogStateJSON(t))))
|
||||
assertStateErrorContains(t, err, "RFC3339")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateCatalogRejectsInvalidOutputRecords(t *testing.T) {
|
||||
tests := map[string]func(*CatalogState){
|
||||
"duplicate path": func(s *CatalogState) {
|
||||
s.Outputs[1].Path = s.Outputs[0].Path
|
||||
},
|
||||
"invalid output path": func(s *CatalogState) {
|
||||
s.Outputs[0].Path = "../report.md"
|
||||
},
|
||||
"invalid pipeline id": func(s *CatalogState) {
|
||||
s.Outputs[0].PipelineID = ".reports"
|
||||
},
|
||||
"invalid destination id": func(s *CatalogState) {
|
||||
s.Outputs[0].DestinationID = ".archive"
|
||||
},
|
||||
"missing source id": func(s *CatalogState) {
|
||||
s.Outputs[0].Source.ID = ""
|
||||
},
|
||||
"invalid source digest": func(s *CatalogState) {
|
||||
s.Outputs[0].Source.Digest = "SHA256:099b205780d2b050024868399961b05731729a548d5d6329c7b06a6740dd75fe"
|
||||
},
|
||||
"missing source created": func(s *CatalogState) {
|
||||
s.Outputs[0].Source.Created = time.Time{}
|
||||
},
|
||||
"invalid kind": func(s *CatalogState) {
|
||||
s.Outputs[0].Kind = "document"
|
||||
},
|
||||
"generated missing source path": func(s *CatalogState) {
|
||||
s.Outputs[1].SourcePath = ""
|
||||
},
|
||||
"generated invalid source path": func(s *CatalogState) {
|
||||
s.Outputs[1].SourcePath = "../report.md"
|
||||
},
|
||||
"generated missing transform": func(s *CatalogState) {
|
||||
s.Outputs[1].Transform = ""
|
||||
},
|
||||
"source output source path": func(s *CatalogState) {
|
||||
s.Outputs[0].SourcePath = "report.md"
|
||||
},
|
||||
"source output transform": func(s *CatalogState) {
|
||||
s.Outputs[0].Transform = "markdown_to_html"
|
||||
},
|
||||
"invalid output digest": func(s *CatalogState) {
|
||||
s.Outputs[0].SHA256 = "SHA256:3640fd37140ee4d2e0e93e78834f232ea67a50e7bc6279203690cc7de1975fa6"
|
||||
},
|
||||
"negative size": func(s *CatalogState) {
|
||||
s.Outputs[0].Size = -1
|
||||
},
|
||||
"invalid url": func(s *CatalogState) {
|
||||
s.Outputs[1].URL = "file:///tmp/report.html"
|
||||
},
|
||||
"missing created at": func(s *CatalogState) {
|
||||
s.Outputs[0].CreatedAt = time.Time{}
|
||||
},
|
||||
"missing updated at": func(s *CatalogState) {
|
||||
s.Outputs[0].UpdatedAt = time.Time{}
|
||||
},
|
||||
}
|
||||
for name, mutate := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
state := validCatalogState(t)
|
||||
mutate(&state)
|
||||
if err := ValidateCatalog(state); err == nil {
|
||||
t.Fatal("ValidateCatalog() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCatalogRejectsForbiddenFields(t *testing.T) {
|
||||
tests := map[string]string{
|
||||
"owners": `"owners": [],`,
|
||||
"sources": `"sources": [],`,
|
||||
"workflow": `"workflow": "additive",`,
|
||||
"source": `"source": {"manifest": {}},`,
|
||||
"manifest": `"manifest": {},`,
|
||||
"pipeline_id": `"pipeline_id": "reports",`,
|
||||
"destination_id": `"destination_id": "archive",`,
|
||||
"published_at": `"published_at": "2026-06-19T12:00:00Z",`,
|
||||
}
|
||||
for name, field := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
body := strings.Replace(validCatalogStateJSON(t), `"created_at":`, field+"\n "+`"created_at":`, 1)
|
||||
_, err := ParseCatalog([]byte(body))
|
||||
assertStateErrorContains(t, err, "unknown field")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCatalogRejectsForbiddenOutputFieldsForSourceOutput(t *testing.T) {
|
||||
tests := map[string]string{
|
||||
"source_path": `"source_path": "report.md",`,
|
||||
"transform": `"transform": "markdown_to_html",`,
|
||||
"empty url": `"url": "",`,
|
||||
}
|
||||
for name, field := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
body := strings.Replace(validCatalogStateJSON(t), `"kind": "source",`, `"kind": "source",`+"\n "+field, 1)
|
||||
_, err := ParseCatalog([]byte(body))
|
||||
if err == nil {
|
||||
t.Fatal("ParseCatalog() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func validCatalogStateJSON(t *testing.T) string {
|
||||
t.Helper()
|
||||
data, err := json.MarshalIndent(validCatalogState(t), "", " ")
|
||||
if err != nil {
|
||||
t.Fatalf("marshal catalog state: %v", err)
|
||||
}
|
||||
return string(data)
|
||||
}
|
||||
|
||||
func catalogStateObject(t *testing.T) map[string]any {
|
||||
t.Helper()
|
||||
var document map[string]any
|
||||
if err := json.Unmarshal([]byte(validCatalogStateJSON(t)), &document); err != nil {
|
||||
t.Fatalf("unmarshal catalog state: %v", err)
|
||||
}
|
||||
return document
|
||||
}
|
||||
|
||||
func firstCatalogOutput(document map[string]any) map[string]any {
|
||||
outputs := document["outputs"].([]any)
|
||||
return outputs[0].(map[string]any)
|
||||
}
|
||||
|
||||
func firstCatalogSource(document map[string]any) map[string]any {
|
||||
return firstCatalogOutput(document)["source"].(map[string]any)
|
||||
}
|
||||
|
||||
func mustMarshalCatalogObject(t *testing.T, document map[string]any) []byte {
|
||||
t.Helper()
|
||||
data, err := json.Marshal(document)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal catalog object: %v", err)
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
func validCatalogState(t *testing.T) CatalogState {
|
||||
t.Helper()
|
||||
manifest := validManifest(t)
|
||||
createdAt := time.Date(2026, 6, 19, 12, 0, 0, 0, time.UTC)
|
||||
updatedAt := time.Date(2026, 6, 19, 12, 5, 0, 0, time.UTC)
|
||||
source := CatalogSourceIdentity{
|
||||
ID: manifest.ID,
|
||||
Digest: manifest.Digest,
|
||||
Created: manifest.Created,
|
||||
}
|
||||
return CatalogState{
|
||||
SchemaVersion: CatalogSchemaVersion,
|
||||
DistributorVersion: "dev",
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: updatedAt,
|
||||
State: StatePolicy{Mode: StateModeCatalog},
|
||||
Outputs: []CatalogOutputFile{{
|
||||
Path: "report.md",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "archive",
|
||||
Source: source,
|
||||
Kind: OutputKindSource,
|
||||
SHA256: manifest.Files[0].SHA256,
|
||||
Size: manifest.Files[0].Size,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: updatedAt,
|
||||
}, {
|
||||
Path: "report.html",
|
||||
PipelineID: "reports",
|
||||
DestinationID: "html",
|
||||
Source: source,
|
||||
Kind: OutputKindGenerated,
|
||||
SourcePath: "report.md",
|
||||
Transform: "markdown_to_html",
|
||||
URL: "https://reports.example.com/report.html",
|
||||
SHA256: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
Size: 128,
|
||||
CreatedAt: createdAt,
|
||||
UpdatedAt: updatedAt,
|
||||
}},
|
||||
}
|
||||
}
|
||||
@@ -23,6 +23,8 @@ const (
|
||||
type DestinationStatus struct {
|
||||
State *DistributorState
|
||||
SharedRoot *SharedRootState
|
||||
Catalog *CatalogState
|
||||
SupersededLegacy *SupersededLegacyState
|
||||
StateErr error
|
||||
HasContents bool
|
||||
}
|
||||
@@ -30,37 +32,75 @@ type DestinationStatus struct {
|
||||
type Comparison struct {
|
||||
Outcome Outcome
|
||||
Reason string
|
||||
Detail ComparisonDetail
|
||||
}
|
||||
|
||||
type ComparisonDetailKind string
|
||||
|
||||
const (
|
||||
ComparisonDetailNone ComparisonDetailKind = ""
|
||||
ComparisonDetailInvalidState ComparisonDetailKind = "invalid_state"
|
||||
ComparisonDetailUnmanagedContent ComparisonDetailKind = "unmanaged_content"
|
||||
ComparisonDetailPipelineIDMismatch ComparisonDetailKind = "pipeline_id_mismatch"
|
||||
ComparisonDetailDestinationIDMismatch ComparisonDetailKind = "destination_id_mismatch"
|
||||
ComparisonDetailDifferentSourceID ComparisonDetailKind = "different_source_id"
|
||||
ComparisonDetailSameCreatedDigestConflict ComparisonDetailKind = "same_created_digest_conflict"
|
||||
ComparisonDetailDestinationNewer ComparisonDetailKind = "destination_newer"
|
||||
ComparisonDetailSharedRootOwnerAbsent ComparisonDetailKind = "shared_root_owner_absent"
|
||||
ComparisonDetailSharedRootOutputOwner ComparisonDetailKind = "shared_root_output_owner_conflict"
|
||||
)
|
||||
|
||||
type ComparisonDetail struct {
|
||||
Kind ComparisonDetailKind
|
||||
CurrentPipelineID string
|
||||
CurrentDestinationID string
|
||||
DestinationPipelineID string
|
||||
DestinationDestinationID string
|
||||
CurrentSourceID string
|
||||
DestinationSourceID string
|
||||
CurrentSourceDigest string
|
||||
DestinationSourceDigest string
|
||||
Path string
|
||||
CurrentOwner OwnerScope
|
||||
ConflictingOwner OwnerScope
|
||||
}
|
||||
|
||||
func CompareSharedRootOwner(source bundle.Manifest, scope OwnerScope, status DestinationStatus) Comparison {
|
||||
if status.StateErr != nil {
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: status.StateErr.Error()}
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: status.StateErr.Error(), Detail: ComparisonDetail{Kind: ComparisonDetailInvalidState}}
|
||||
}
|
||||
if status.SharedRoot != nil {
|
||||
if err := ValidateSharedRoot(*status.SharedRoot); err != nil {
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: err.Error()}
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: err.Error(), Detail: ComparisonDetail{Kind: ComparisonDetailInvalidState}}
|
||||
}
|
||||
owner, ok := status.SharedRoot.Owner(scope)
|
||||
if !ok {
|
||||
return Comparison{Outcome: OutcomeDestinationAbsent, Reason: fmt.Sprintf("destination owner %s/%s is absent", scope.PipelineID, scope.DestinationID)}
|
||||
return Comparison{
|
||||
Outcome: OutcomeDestinationAbsent,
|
||||
Reason: fmt.Sprintf("destination owner %s/%s is absent", scope.PipelineID, scope.DestinationID),
|
||||
Detail: ComparisonDetail{
|
||||
Kind: ComparisonDetailSharedRootOwnerAbsent,
|
||||
CurrentOwner: scope,
|
||||
},
|
||||
}
|
||||
}
|
||||
return compareManifests(source, owner.Source.Manifest)
|
||||
}
|
||||
if status.State != nil {
|
||||
destinationState := *status.State
|
||||
if err := Validate(destinationState); err != nil {
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: err.Error()}
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: err.Error(), Detail: ComparisonDetail{Kind: ComparisonDetailInvalidState}}
|
||||
}
|
||||
if destinationState.PipelineID != scope.PipelineID {
|
||||
return Comparison{Outcome: OutcomeIdentityMismatch, Reason: fmt.Sprintf("pipeline id %q does not match %q", destinationState.PipelineID, scope.PipelineID)}
|
||||
return pipelineIDMismatchComparison(destinationState.PipelineID, scope.PipelineID)
|
||||
}
|
||||
if destinationState.DestinationID != scope.DestinationID {
|
||||
return Comparison{Outcome: OutcomeIdentityMismatch, Reason: fmt.Sprintf("destination id %q does not match %q", destinationState.DestinationID, scope.DestinationID)}
|
||||
return destinationIDMismatchComparison(destinationState.DestinationID, scope.DestinationID)
|
||||
}
|
||||
return compareManifests(source, destinationState.Source.Manifest)
|
||||
}
|
||||
if status.HasContents {
|
||||
return Comparison{Outcome: OutcomeDestinationUnmanaged, Reason: "destination has content but no distributor state"}
|
||||
return Comparison{Outcome: OutcomeDestinationUnmanaged, Reason: "destination has content but no distributor state", Detail: ComparisonDetail{Kind: ComparisonDetailUnmanagedContent}}
|
||||
}
|
||||
return Comparison{Outcome: OutcomeDestinationAbsent, Reason: "destination state is absent"}
|
||||
}
|
||||
@@ -70,45 +110,95 @@ func compareManifests(source, destination bundle.Manifest) Comparison {
|
||||
return Comparison{Outcome: OutcomeSameSource, Reason: "destination source manifest matches source"}
|
||||
}
|
||||
if destination.ID != source.ID {
|
||||
return Comparison{Outcome: OutcomeDifferentSourceConflict, Reason: "destination source id differs from source"}
|
||||
return Comparison{
|
||||
Outcome: OutcomeDifferentSourceConflict,
|
||||
Reason: "destination source id differs from source",
|
||||
Detail: ComparisonDetail{
|
||||
Kind: ComparisonDetailDifferentSourceID,
|
||||
CurrentSourceID: source.ID,
|
||||
DestinationSourceID: destination.ID,
|
||||
},
|
||||
}
|
||||
}
|
||||
if destination.Created.Before(source.Created) {
|
||||
return Comparison{Outcome: OutcomeDestinationOlder, Reason: "destination source is older than source"}
|
||||
}
|
||||
if destination.Created.After(source.Created) {
|
||||
return Comparison{Outcome: OutcomeDestinationNewer, Reason: "destination source is newer than source"}
|
||||
return Comparison{
|
||||
Outcome: OutcomeDestinationNewer,
|
||||
Reason: "destination source is newer than source",
|
||||
Detail: ComparisonDetail{
|
||||
Kind: ComparisonDetailDestinationNewer,
|
||||
CurrentSourceID: source.ID,
|
||||
DestinationSourceID: destination.ID,
|
||||
},
|
||||
}
|
||||
}
|
||||
if destination.Digest != source.Digest {
|
||||
return Comparison{Outcome: OutcomeSameCreatedConflict, Reason: "destination source has same id and created time but different digest"}
|
||||
return Comparison{
|
||||
Outcome: OutcomeSameCreatedConflict,
|
||||
Reason: "destination source has same id and created time but different digest",
|
||||
Detail: ComparisonDetail{
|
||||
Kind: ComparisonDetailSameCreatedDigestConflict,
|
||||
CurrentSourceID: source.ID,
|
||||
DestinationSourceID: destination.ID,
|
||||
CurrentSourceDigest: source.Digest,
|
||||
DestinationSourceDigest: destination.Digest,
|
||||
},
|
||||
}
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: "destination source differs from source without a supported comparison outcome"}
|
||||
}
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: "destination source differs from source without a supported comparison outcome", Detail: ComparisonDetail{Kind: ComparisonDetailInvalidState}}
|
||||
}
|
||||
|
||||
func Compare(source bundle.Manifest, pipelineID, destinationID string, status DestinationStatus) Comparison {
|
||||
if status.StateErr != nil {
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: status.StateErr.Error()}
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: status.StateErr.Error(), Detail: ComparisonDetail{Kind: ComparisonDetailInvalidState}}
|
||||
}
|
||||
if status.State == nil {
|
||||
if status.HasContents {
|
||||
return Comparison{Outcome: OutcomeDestinationUnmanaged, Reason: "destination has content but no distributor state"}
|
||||
return Comparison{Outcome: OutcomeDestinationUnmanaged, Reason: "destination has content but no distributor state", Detail: ComparisonDetail{Kind: ComparisonDetailUnmanagedContent}}
|
||||
}
|
||||
return Comparison{Outcome: OutcomeDestinationAbsent, Reason: "destination state is absent"}
|
||||
}
|
||||
|
||||
destinationState := *status.State
|
||||
if err := Validate(destinationState); err != nil {
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: err.Error()}
|
||||
return Comparison{Outcome: OutcomeInvalidState, Reason: err.Error(), Detail: ComparisonDetail{Kind: ComparisonDetailInvalidState}}
|
||||
}
|
||||
if destinationState.PipelineID != pipelineID {
|
||||
return Comparison{Outcome: OutcomeIdentityMismatch, Reason: fmt.Sprintf("pipeline id %q does not match %q", destinationState.PipelineID, pipelineID)}
|
||||
return pipelineIDMismatchComparison(destinationState.PipelineID, pipelineID)
|
||||
}
|
||||
if destinationState.DestinationID != destinationID {
|
||||
return Comparison{Outcome: OutcomeIdentityMismatch, Reason: fmt.Sprintf("destination id %q does not match %q", destinationState.DestinationID, destinationID)}
|
||||
return destinationIDMismatchComparison(destinationState.DestinationID, destinationID)
|
||||
}
|
||||
|
||||
return compareManifests(source, destinationState.Source.Manifest)
|
||||
}
|
||||
|
||||
func pipelineIDMismatchComparison(destinationPipelineID, currentPipelineID string) Comparison {
|
||||
return Comparison{
|
||||
Outcome: OutcomeIdentityMismatch,
|
||||
Reason: fmt.Sprintf("pipeline id %q does not match %q", destinationPipelineID, currentPipelineID),
|
||||
Detail: ComparisonDetail{
|
||||
Kind: ComparisonDetailPipelineIDMismatch,
|
||||
CurrentPipelineID: currentPipelineID,
|
||||
DestinationPipelineID: destinationPipelineID,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func destinationIDMismatchComparison(destinationDestinationID, currentDestinationID string) Comparison {
|
||||
return Comparison{
|
||||
Outcome: OutcomeIdentityMismatch,
|
||||
Reason: fmt.Sprintf("destination id %q does not match %q", destinationDestinationID, currentDestinationID),
|
||||
Detail: ComparisonDetail{
|
||||
Kind: ComparisonDetailDestinationIDMismatch,
|
||||
CurrentDestinationID: currentDestinationID,
|
||||
DestinationDestinationID: destinationDestinationID,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func manifestsEqual(a, b bundle.Manifest) bool {
|
||||
if a.SchemaVersion != b.SchemaVersion ||
|
||||
a.ID != b.ID ||
|
||||
|
||||
@@ -97,6 +97,93 @@ func TestCompareOutcomes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompareReportsStructuredDetails(t *testing.T) {
|
||||
source := validManifest(t)
|
||||
tests := []struct {
|
||||
name string
|
||||
status DestinationStatus
|
||||
wantKind ComparisonDetailKind
|
||||
assertions func(t *testing.T, detail ComparisonDetail)
|
||||
}{
|
||||
{
|
||||
name: "pipeline mismatch",
|
||||
status: DestinationStatus{State: withState(t, source, func(s *DistributorState) { s.PipelineID = "other" })},
|
||||
wantKind: ComparisonDetailPipelineIDMismatch,
|
||||
assertions: func(t *testing.T, detail ComparisonDetail) {
|
||||
t.Helper()
|
||||
if detail.DestinationPipelineID != "other" || detail.CurrentPipelineID != "reports" {
|
||||
t.Fatalf("detail = %#v, want pipeline ids", detail)
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "destination mismatch",
|
||||
status: DestinationStatus{State: withState(t, source, func(s *DistributorState) { s.DestinationID = "other" })},
|
||||
wantKind: ComparisonDetailDestinationIDMismatch,
|
||||
assertions: func(t *testing.T, detail ComparisonDetail) {
|
||||
t.Helper()
|
||||
if detail.DestinationDestinationID != "other" || detail.CurrentDestinationID != "archive" {
|
||||
t.Fatalf("detail = %#v, want destination ids", detail)
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "different source id",
|
||||
status: DestinationStatus{State: withState(t, source, func(s *DistributorState) { s.Source.Manifest.ID = "other.source" })},
|
||||
wantKind: ComparisonDetailDifferentSourceID,
|
||||
assertions: func(t *testing.T, detail ComparisonDetail) {
|
||||
t.Helper()
|
||||
if detail.DestinationSourceID != "other.source" || detail.CurrentSourceID != source.ID {
|
||||
t.Fatalf("detail = %#v, want source ids", detail)
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "same created digest conflict",
|
||||
status: DestinationStatus{State: withState(t, source, func(s *DistributorState) {
|
||||
s.Source.Manifest.Files[0].SHA256 = "sha256:3333333333333333333333333333333333333333333333333333333333333333"
|
||||
s.Source.Manifest.Digest = bundle.BundleDigest(s.Source.Manifest.Files)
|
||||
})},
|
||||
wantKind: ComparisonDetailSameCreatedDigestConflict,
|
||||
assertions: func(t *testing.T, detail ComparisonDetail) {
|
||||
t.Helper()
|
||||
if detail.CurrentSourceDigest == "" || detail.DestinationSourceDigest == "" || detail.CurrentSourceDigest == detail.DestinationSourceDigest {
|
||||
t.Fatalf("detail = %#v, want different source digests", detail)
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "destination newer",
|
||||
status: DestinationStatus{State: withState(t, source, func(s *DistributorState) {
|
||||
s.Source.Manifest.Created = source.Created.Add(time.Hour)
|
||||
})},
|
||||
wantKind: ComparisonDetailDestinationNewer,
|
||||
},
|
||||
{
|
||||
name: "invalid state",
|
||||
status: DestinationStatus{StateErr: errors.New("invalid json")},
|
||||
wantKind: ComparisonDetailInvalidState,
|
||||
},
|
||||
{
|
||||
name: "unmanaged content",
|
||||
status: DestinationStatus{HasContents: true},
|
||||
wantKind: ComparisonDetailUnmanagedContent,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := Compare(source, "reports", "archive", tt.status)
|
||||
if got.Detail.Kind != tt.wantKind {
|
||||
t.Fatalf("Compare() detail kind = %q, want %q; comparison=%#v", got.Detail.Kind, tt.wantKind, got)
|
||||
}
|
||||
if tt.assertions != nil {
|
||||
tt.assertions(t, got.Detail)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func withState(t *testing.T, source bundle.Manifest, mutate func(*DistributorState)) *DistributorState {
|
||||
t.Helper()
|
||||
stateManifest := source
|
||||
|
||||
@@ -14,9 +14,11 @@ import (
|
||||
const (
|
||||
SchemaVersion = 2
|
||||
SharedRootSchemaVersion = 3
|
||||
CatalogSchemaVersion = 4
|
||||
legacySchemaVersion = 1
|
||||
StateModeSingleOwner = config.StateModeSingleOwner
|
||||
StateModeSharedRoot = config.StateModeSharedRoot
|
||||
StateModeCatalog = "catalog"
|
||||
)
|
||||
|
||||
type DistributorState struct {
|
||||
|
||||
@@ -26,6 +26,15 @@ func FindOutputByPath(outputs []OutputFile, path string) (OutputFile, bool) {
|
||||
return OutputFile{}, false
|
||||
}
|
||||
|
||||
func FindCatalogOutputByPath(outputs []CatalogOutputFile, path string) (CatalogOutputFile, bool) {
|
||||
for _, output := range outputs {
|
||||
if output.Path == path {
|
||||
return output, true
|
||||
}
|
||||
}
|
||||
return CatalogOutputFile{}, false
|
||||
}
|
||||
|
||||
func MergeOutputFiles(retained, planned []OutputFile) ([]OutputFile, error) {
|
||||
outputs := make([]OutputFile, 0, len(retained)+len(planned))
|
||||
indexByPath := make(map[string]int, len(retained)+len(planned))
|
||||
@@ -178,6 +187,62 @@ func RemoveMissingSharedRootOutputs(s SharedRootState, missingPaths []string) (S
|
||||
return next, changed
|
||||
}
|
||||
|
||||
func CatalogOutputsForOwner(outputs []CatalogOutputFile, scope OwnerScope) []CatalogOutputFile {
|
||||
selected := make([]CatalogOutputFile, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
if output.PipelineID == scope.PipelineID && output.DestinationID == scope.DestinationID {
|
||||
selected = append(selected, output)
|
||||
}
|
||||
}
|
||||
return selected
|
||||
}
|
||||
|
||||
func CatalogManagedOutputPaths(s CatalogState) []string {
|
||||
paths := make([]string, 0, len(s.Outputs))
|
||||
for _, output := range s.Outputs {
|
||||
paths = append(paths, output.Path)
|
||||
}
|
||||
return paths
|
||||
}
|
||||
|
||||
func RemoveMissingCatalogOwnerOutputs(s CatalogState, scope OwnerScope, missingPaths []string) (CatalogState, bool) {
|
||||
if len(missingPaths) == 0 {
|
||||
return s, false
|
||||
}
|
||||
missing := pathSet(missingPaths)
|
||||
next := s
|
||||
next.Outputs = make([]CatalogOutputFile, 0, len(s.Outputs))
|
||||
changed := false
|
||||
for _, output := range s.Outputs {
|
||||
if output.PipelineID == scope.PipelineID && output.DestinationID == scope.DestinationID {
|
||||
if _, remove := missing[output.Path]; remove {
|
||||
changed = true
|
||||
continue
|
||||
}
|
||||
}
|
||||
next.Outputs = append(next.Outputs, output)
|
||||
}
|
||||
return next, changed
|
||||
}
|
||||
|
||||
func RemoveMissingCatalogOutputs(s CatalogState, missingPaths []string) (CatalogState, bool) {
|
||||
if len(missingPaths) == 0 {
|
||||
return s, false
|
||||
}
|
||||
missing := pathSet(missingPaths)
|
||||
next := s
|
||||
next.Outputs = make([]CatalogOutputFile, 0, len(s.Outputs))
|
||||
changed := false
|
||||
for _, output := range s.Outputs {
|
||||
if _, remove := missing[output.Path]; remove {
|
||||
changed = true
|
||||
continue
|
||||
}
|
||||
next.Outputs = append(next.Outputs, output)
|
||||
}
|
||||
return next, changed
|
||||
}
|
||||
|
||||
func (s SharedRootState) OutputOwner(path string) (OwnerScope, bool) {
|
||||
for _, output := range s.Outputs {
|
||||
if output.Path == path {
|
||||
@@ -191,7 +256,17 @@ func (s SharedRootState) PathOwnershipConflict(scope OwnerScope, paths []string)
|
||||
for _, path := range paths {
|
||||
owner, exists := s.OutputOwner(path)
|
||||
if exists && owner != scope {
|
||||
return PathOwnershipConflict{Path: path, Owner: owner}, true
|
||||
return PathOwnershipConflict{
|
||||
Path: path,
|
||||
Owner: owner,
|
||||
CurrentOwner: scope,
|
||||
Detail: ComparisonDetail{
|
||||
Kind: ComparisonDetailSharedRootOutputOwner,
|
||||
Path: path,
|
||||
CurrentOwner: scope,
|
||||
ConflictingOwner: owner,
|
||||
},
|
||||
}, true
|
||||
}
|
||||
}
|
||||
return PathOwnershipConflict{}, false
|
||||
|
||||
@@ -49,6 +49,22 @@ func SharedRootPruneCandidates(s SharedRootState, scope OwnerScope) []PruneCandi
|
||||
return candidates
|
||||
}
|
||||
|
||||
func CatalogPruneCandidates(s CatalogState, scope OwnerScope) []PruneCandidate {
|
||||
candidates := make([]PruneCandidate, 0, len(s.Outputs))
|
||||
for _, output := range s.Outputs {
|
||||
if output.PipelineID != scope.PipelineID || output.DestinationID != scope.DestinationID {
|
||||
continue
|
||||
}
|
||||
owner := scope
|
||||
candidates = append(candidates, PruneCandidate{
|
||||
Path: output.Path,
|
||||
UpdatedAt: output.UpdatedAt,
|
||||
Owner: &owner,
|
||||
})
|
||||
}
|
||||
return candidates
|
||||
}
|
||||
|
||||
func PlanPrune(candidates []PruneCandidate, options PrunePlanOptions) PrunePlan {
|
||||
ordered := append([]PruneCandidate(nil), candidates...)
|
||||
sortPruneCandidatesNewestFirst(ordered)
|
||||
|
||||
@@ -16,6 +16,12 @@ import (
|
||||
type StateDocument struct {
|
||||
SingleOwner *DistributorState
|
||||
SharedRoot *SharedRootState
|
||||
Catalog *CatalogState
|
||||
SupersededLegacy *SupersededLegacyState
|
||||
}
|
||||
|
||||
type SupersededLegacyState struct {
|
||||
SchemaVersion int
|
||||
}
|
||||
|
||||
type SharedRootState struct {
|
||||
@@ -59,6 +65,8 @@ type SharedRootOutputFile struct {
|
||||
type PathOwnershipConflict struct {
|
||||
Path string
|
||||
Owner OwnerScope
|
||||
CurrentOwner OwnerScope
|
||||
Detail ComparisonDetail
|
||||
}
|
||||
|
||||
type rawSharedRootState struct {
|
||||
@@ -101,18 +109,18 @@ func ParseDocument(data []byte) (StateDocument, error) {
|
||||
if err != nil {
|
||||
return StateDocument{}, err
|
||||
}
|
||||
if schemaVersion == SharedRootSchemaVersion {
|
||||
sharedRoot, err := ParseSharedRoot(data)
|
||||
switch schemaVersion {
|
||||
case legacySchemaVersion, SchemaVersion, SharedRootSchemaVersion:
|
||||
return StateDocument{SupersededLegacy: &SupersededLegacyState{SchemaVersion: schemaVersion}}, nil
|
||||
case CatalogSchemaVersion:
|
||||
catalog, err := ParseCatalog(data)
|
||||
if err != nil {
|
||||
return StateDocument{}, err
|
||||
}
|
||||
return StateDocument{SharedRoot: &sharedRoot}, nil
|
||||
return StateDocument{Catalog: &catalog}, nil
|
||||
default:
|
||||
return StateDocument{}, fmt.Errorf("state schema_version %d is unsupported", schemaVersion)
|
||||
}
|
||||
singleOwner, err := Parse(data)
|
||||
if err != nil {
|
||||
return StateDocument{}, err
|
||||
}
|
||||
return StateDocument{SingleOwner: &singleOwner}, nil
|
||||
}
|
||||
|
||||
func parseSchemaVersion(data []byte) (int, error) {
|
||||
@@ -123,6 +131,10 @@ func parseSchemaVersion(data []byte) (int, error) {
|
||||
if err := decoder.Decode(&raw); err != nil {
|
||||
return 0, fmt.Errorf("parse distributor state: %w", err)
|
||||
}
|
||||
var extra any
|
||||
if err := decoder.Decode(&extra); err != io.EOF {
|
||||
return 0, fmt.Errorf("parse distributor state: trailing data")
|
||||
}
|
||||
if raw.SchemaVersion == nil {
|
||||
return 0, fmt.Errorf("state schema_version is required")
|
||||
}
|
||||
|
||||
@@ -10,24 +10,6 @@ import (
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||
)
|
||||
|
||||
func TestParseDocumentHandlesSingleOwnerAndSharedRoot(t *testing.T) {
|
||||
singleOwner, err := ParseDocument([]byte(validStateJSON(t)))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseDocument(single owner) error = %v", err)
|
||||
}
|
||||
if singleOwner.SingleOwner == nil || singleOwner.SharedRoot != nil {
|
||||
t.Fatalf("single owner document = %#v", singleOwner)
|
||||
}
|
||||
|
||||
sharedRoot, err := ParseDocument([]byte(validSharedRootStateJSON(t)))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseDocument(shared root) error = %v", err)
|
||||
}
|
||||
if sharedRoot.SharedRoot == nil || sharedRoot.SingleOwner != nil {
|
||||
t.Fatalf("shared root document = %#v", sharedRoot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseSharedRootState(t *testing.T) {
|
||||
state, err := ParseSharedRoot([]byte(validSharedRootStateJSON(t)))
|
||||
if err != nil {
|
||||
@@ -130,6 +112,12 @@ func TestSharedRootOutputHelpers(t *testing.T) {
|
||||
if !ok || conflict.Owner != html {
|
||||
t.Fatalf("conflict = %#v ok=%t, want html owner conflict", conflict, ok)
|
||||
}
|
||||
if conflict.Detail.Kind != ComparisonDetailSharedRootOutputOwner {
|
||||
t.Fatalf("conflict detail kind = %q, want %q", conflict.Detail.Kind, ComparisonDetailSharedRootOutputOwner)
|
||||
}
|
||||
if conflict.Detail.Path != "report.html" || conflict.Detail.CurrentOwner != archive || conflict.Detail.ConflictingOwner != html {
|
||||
t.Fatalf("conflict detail = %#v, want path and owners", conflict.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRemoveMissingSharedRootOwnerOutputs(t *testing.T) {
|
||||
@@ -241,6 +229,12 @@ func TestCompareSharedRootOwnerScopesCurrentOwner(t *testing.T) {
|
||||
if missing.Outcome != OutcomeDestinationAbsent {
|
||||
t.Fatalf("missing owner comparison = %#v, want destination absent", missing)
|
||||
}
|
||||
if missing.Detail.Kind != ComparisonDetailSharedRootOwnerAbsent {
|
||||
t.Fatalf("missing owner detail kind = %q, want %q", missing.Detail.Kind, ComparisonDetailSharedRootOwnerAbsent)
|
||||
}
|
||||
if missing.Detail.CurrentOwner != CurrentOwnerScope("missing", "archive") {
|
||||
t.Fatalf("missing owner detail = %#v, want missing/archive", missing.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompareSharedRootOwnerAcceptsMatchingSingleOwnerState(t *testing.T) {
|
||||
|
||||
@@ -123,7 +123,7 @@ func (t *Transformer) render(ctx context.Context, req transform.Request, sourceF
|
||||
if err := t.renderer.Convert(data, &rendered); err != nil {
|
||||
return nil, fmt.Errorf("render markdown source %q: %w", sourceFile, err)
|
||||
}
|
||||
return wrapHTML(rendered.Bytes()), nil
|
||||
return wrapHTML(rendered.Bytes(), req.Markdown.CssHref), nil
|
||||
}
|
||||
|
||||
func markdownMode(mode string) string {
|
||||
|
||||
@@ -40,6 +40,43 @@ func TestGenerateMarkdownSidecar(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownWithoutCSSHrefPreservesWrapper(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
want := "<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title></title>\n</head>\n<body>\n<h1>Title</h1>\n<p>Hello.</p>\n</body>\n</html>\n"
|
||||
if got := string(outputs[0].Data); got != want {
|
||||
t.Fatalf("html = %q, want existing wrapper %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownSidecarWithCSSHref(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
outputs, err := New().Generate(context.Background(), transform.Request{
|
||||
SourceBackend: backend,
|
||||
SourceBundle: sourceBundle,
|
||||
Markdown: transform.MarkdownOptions{CssHref: "/assets/report.css?v=1&theme=main"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
output := outputs[0]
|
||||
html := string(output.Data)
|
||||
wantLink := "<meta charset=\"utf-8\">\n<link rel=\"stylesheet\" href=\"/assets/report.css?v=1&theme=main\">\n<title></title>"
|
||||
if !strings.Contains(html, wantLink) {
|
||||
t.Fatalf("html = %q, want stylesheet link %q", html, wantLink)
|
||||
}
|
||||
if output.SHA256 != bundle.FileDigest(output.Data) || output.Size != int64(len(output.Data)) {
|
||||
t.Fatalf("digest/size metadata = %s/%d", output.SHA256, output.Size)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownIndexExplicitInput(t *testing.T) {
|
||||
backend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, backend, "", testutil.BundleOptions{
|
||||
@@ -74,6 +111,29 @@ func TestGenerateMarkdownIndexExplicitInput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownIndexWithCSSHref(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
outputs, err := New().Generate(context.Background(), transform.Request{
|
||||
SourceBackend: backend,
|
||||
SourceBundle: sourceBundle,
|
||||
Markdown: transform.MarkdownOptions{
|
||||
Mode: transform.MarkdownModeIndex,
|
||||
CssHref: "https://example.com/assets/report.css",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
if got, want := outputs[0].Path, "index.html"; got != want {
|
||||
t.Fatalf("path = %q, want %q", got, want)
|
||||
}
|
||||
if !strings.Contains(string(outputs[0].Data), `<link rel="stylesheet" href="https://example.com/assets/report.css">`) {
|
||||
t.Fatalf("html = %q, want stylesheet link", outputs[0].Data)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownIndexSelectsOnlyMarkdownFile(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
|
||||
@@ -1,10 +1,19 @@
|
||||
package markdown
|
||||
|
||||
import "bytes"
|
||||
import (
|
||||
"bytes"
|
||||
"html"
|
||||
)
|
||||
|
||||
func wrapHTML(body []byte) []byte {
|
||||
func wrapHTML(body []byte, cssHref string) []byte {
|
||||
var buf bytes.Buffer
|
||||
buf.WriteString("<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title></title>\n</head>\n<body>\n")
|
||||
buf.WriteString("<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n")
|
||||
if cssHref != "" {
|
||||
buf.WriteString("<link rel=\"stylesheet\" href=\"")
|
||||
buf.WriteString(html.EscapeString(cssHref))
|
||||
buf.WriteString("\">\n")
|
||||
}
|
||||
buf.WriteString("<title></title>\n</head>\n<body>\n")
|
||||
buf.Write(body)
|
||||
buf.WriteString("</body>\n</html>\n")
|
||||
return buf.Bytes()
|
||||
|
||||
@@ -25,6 +25,7 @@ type Request struct {
|
||||
type MarkdownOptions struct {
|
||||
Mode string
|
||||
Input string
|
||||
CssHref string
|
||||
}
|
||||
|
||||
type Transformer interface {
|
||||
|
||||
Reference in New Issue
Block a user