Add destination link metadata
This commit is contained in:
@@ -186,7 +186,7 @@ Command-specific JSON results:
|
|||||||
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
|
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
|
||||||
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
|
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
|
||||||
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
|
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
|
||||||
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, optional fixed path mapping markers, output records, final counters, warnings, and partial failure records.
|
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, path mapping markers, optional primary URLs, output records with optional URLs, final counters, warnings, and partial failure records.
|
||||||
|
|
||||||
## Diagnostics
|
## Diagnostics
|
||||||
|
|
||||||
|
|||||||
@@ -117,6 +117,28 @@ Fixed destinations select the newest discovered source bundle by manifest `creat
|
|||||||
|
|
||||||
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
|
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
|
||||||
|
|
||||||
|
## Destination Links
|
||||||
|
|
||||||
|
Destinations can record public URLs for published outputs:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
links:
|
||||||
|
base_url: https://reports.example.com/archive
|
||||||
|
primary: auto
|
||||||
|
```
|
||||||
|
|
||||||
|
`links.base_url` is an absolute `http` or `https` URL corresponding to the destination backend root. It may include a path prefix, but it must not include a query string or fragment. Distributor does not infer public URLs from backend config.
|
||||||
|
|
||||||
|
`links.primary` selects the top-level primary URL stored in destination state:
|
||||||
|
|
||||||
|
- `auto`: prefer `index.html`, then generated HTML, then source outputs.
|
||||||
|
- `html`: use the first generated HTML output.
|
||||||
|
- `source`: use the first copied source output.
|
||||||
|
|
||||||
|
If a destination has no `links` block, no URL metadata is generated. If a primary policy has no matching output, per-output URLs are still recorded and the top-level primary URL is omitted.
|
||||||
|
|
||||||
|
Output URLs are built from `links.base_url`, the destination bundle path, and the output path using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
|
||||||
|
|
||||||
## Reference
|
## Reference
|
||||||
|
|
||||||
Top level:
|
Top level:
|
||||||
@@ -156,6 +178,8 @@ Destination:
|
|||||||
- `publish`: optional; defaults to source-only publication.
|
- `publish`: optional; defaults to source-only publication.
|
||||||
- `transform`: required only for generated HTML publication.
|
- `transform`: required only for generated HTML publication.
|
||||||
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
|
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
|
||||||
|
- `links.base_url`: optional links block; when present, `base_url` is required and must be an absolute HTTP or HTTPS URL without query string or fragment.
|
||||||
|
- `links.primary`: optional; defaults to `auto`. Accepted values are `auto`, `html`, and `source`.
|
||||||
- `transfer`: optional; defaults described below.
|
- `transfer`: optional; defaults described below.
|
||||||
|
|
||||||
Accepted backend names:
|
Accepted backend names:
|
||||||
@@ -246,6 +270,7 @@ Defaults are applied after YAML decoding and before validation:
|
|||||||
- `publish.source: true`
|
- `publish.source: true`
|
||||||
- `publish.html: false`
|
- `publish.html: false`
|
||||||
- `path_mapping.mode: preserve_relative`
|
- `path_mapping.mode: preserve_relative`
|
||||||
|
- `links.primary: auto` when a `links` block is present and `primary` is omitted
|
||||||
- `transfer.on_destination_same: skip`
|
- `transfer.on_destination_same: skip`
|
||||||
- `transfer.on_destination_older: replace`
|
- `transfer.on_destination_older: replace`
|
||||||
- `transfer.on_destination_newer: skip`
|
- `transfer.on_destination_newer: skip`
|
||||||
|
|||||||
@@ -6,9 +6,9 @@
|
|||||||
|
|
||||||
## Inputs and outputs
|
## Inputs and outputs
|
||||||
|
|
||||||
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, writes summary output when stdout is supplied, and returns an aggregated error if any destination fails.
|
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, output format, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, writes text or JSON output when stdout is supplied, and returns an aggregated error if any destination fails.
|
||||||
|
|
||||||
`Validate` and `Inspect` accept a local path. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
|
`Validate` and `Inspect` accept either a local path or one configured pipeline source. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
|
||||||
|
|
||||||
## Run flow
|
## Run flow
|
||||||
|
|
||||||
@@ -17,11 +17,12 @@ The runner:
|
|||||||
1. loads config from the supplied path or `config.DefaultConfigPath`;
|
1. loads config from the supplied path or `config.DefaultConfigPath`;
|
||||||
2. opens the configured source backend;
|
2. opens the configured source backend;
|
||||||
3. discovers validated bundles from the source root;
|
3. discovers validated bundles from the source root;
|
||||||
4. opens each destination backend independently;
|
4. selects source bundles for each destination according to destination path mapping;
|
||||||
5. builds a publish plan for each bundle and destination;
|
5. opens each destination backend independently;
|
||||||
6. prints plan lines and records summary counters;
|
6. builds publish plans for the selected bundle and destination combinations;
|
||||||
7. executes publish or replacement plans unless dry-run is enabled;
|
7. prints plan lines or JSON action records and records summary counters;
|
||||||
8. invokes the notifier after successful publish or replacement actions.
|
8. executes publish or replacement plans unless dry-run is enabled;
|
||||||
|
9. invokes the notifier after successful publish or replacement actions.
|
||||||
|
|
||||||
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out.
|
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out.
|
||||||
|
|
||||||
@@ -47,7 +48,7 @@ Stdout write errors are returned immediately because the caller's requested outp
|
|||||||
|
|
||||||
`internal/app` coordinates packages but does not own manifest validation rules, destination state comparison, storage path rules, output planning, transform rendering, or backend-specific filesystem behavior.
|
`internal/app` coordinates packages but does not own manifest validation rules, destination state comparison, storage path rules, output planning, transform rendering, or backend-specific filesystem behavior.
|
||||||
|
|
||||||
`Validate` and `Inspect` are local path commands. Remote execution wiring currently belongs to `Run`.
|
Configured-source `Validate` and `Inspect` share source backend construction with `Run` and do not open destinations.
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ Defaults are applied in `ApplyDefaults`:
|
|||||||
- destination publish policy defaults to source output only;
|
- destination publish policy defaults to source output only;
|
||||||
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
||||||
- destination path mapping defaults to `preserve_relative`;
|
- destination path mapping defaults to `preserve_relative`;
|
||||||
|
- destination link primary policy defaults to `auto` when a `links` block is present;
|
||||||
- `transfer.on_destination_same` defaults to `skip`;
|
- `transfer.on_destination_same` defaults to `skip`;
|
||||||
- `transfer.on_destination_older` defaults to `replace`;
|
- `transfer.on_destination_older` defaults to `replace`;
|
||||||
- `transfer.on_destination_newer` defaults to `skip`;
|
- `transfer.on_destination_newer` defaults to `skip`;
|
||||||
@@ -33,7 +34,7 @@ Defaults are applied in `ApplyDefaults`:
|
|||||||
|
|
||||||
## Validation responsibilities
|
## Validation responsibilities
|
||||||
|
|
||||||
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, and valid transfer actions.
|
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
|
||||||
|
|
||||||
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
||||||
|
|
||||||
@@ -41,6 +42,8 @@ Transfer validation accepts `replace` for `on_destination_newer` and `on_conflic
|
|||||||
|
|
||||||
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
|
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
|
||||||
|
|
||||||
|
Destination links are optional. When a `links` block is present, `base_url` is required, must use `http` or `https`, and must not include a query string or fragment. `primary` accepts `auto`, `html`, and `source`.
|
||||||
|
|
||||||
## Executable support boundary
|
## Executable support boundary
|
||||||
|
|
||||||
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
|
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
|
||||||
|
|||||||
@@ -6,9 +6,9 @@
|
|||||||
|
|
||||||
## Inputs and outputs
|
## Inputs and outputs
|
||||||
|
|
||||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
|
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
|
||||||
|
|
||||||
Output is a plan with an action, reason, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
|
Output is a plan with an action, reason, optional primary URL, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
|
||||||
|
|
||||||
## Actions
|
## Actions
|
||||||
|
|
||||||
@@ -28,6 +28,8 @@ The package uses `internal/state` for destination comparison, `internal/storage`
|
|||||||
|
|
||||||
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
|
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
|
||||||
|
|
||||||
|
When link config is present, publish planning builds per-output URLs from `links.base_url`, the destination bundle path, and each output path. `index.html` outputs use directory-style URLs. The primary URL is selected from planned outputs according to the destination primary policy.
|
||||||
|
|
||||||
## Safety
|
## Safety
|
||||||
|
|
||||||
Normal replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path before writing outputs and state. Failed writes trigger cleanup of outputs written during the failed attempt where practical.
|
Normal replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path before writing outputs and state. Failed writes trigger cleanup of outputs written during the failed attempt where practical.
|
||||||
@@ -40,6 +42,7 @@ Before changing publish behavior, inspect tests under `internal/publish` and run
|
|||||||
|
|
||||||
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
|
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
|
||||||
- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
|
- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
|
||||||
|
- URL generation uses URL path semantics and does not infer public URLs from backend configuration.
|
||||||
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
||||||
- Forced replacement is explicit per run and deletes only within the destination bundle path.
|
- Forced replacement is explicit per run and deletes only within the destination bundle path.
|
||||||
- Publish execution writes destination state after selected outputs are written.
|
- Publish execution writes destination state after selected outputs are written.
|
||||||
|
|||||||
@@ -19,13 +19,15 @@ Input is JSON destination state plus the current source manifest, pipeline id, d
|
|||||||
- `source.manifest`
|
- `source.manifest`
|
||||||
- `outputs`
|
- `outputs`
|
||||||
|
|
||||||
`distributor_version` is optional diagnostic metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
|
`distributor_version` is optional diagnostic metadata. `links` is optional URL metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
|
||||||
|
|
||||||
The embedded `source.manifest` is validated with the same source manifest rules used by `internal/bundle`.
|
The embedded `source.manifest` is validated with the same source manifest rules used by `internal/bundle`.
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`.
|
Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`. Outputs may record `url` when the destination has link generation configured.
|
||||||
|
|
||||||
|
The optional top-level `links.primary_url` records the selected primary URL for the published destination bundle. It is omitted when link generation is not configured or when the destination primary policy has no matching output.
|
||||||
|
|
||||||
## Comparison
|
## Comparison
|
||||||
|
|
||||||
@@ -33,7 +35,7 @@ Comparison outcomes cover absent destination state, unmanaged destination conten
|
|||||||
|
|
||||||
## Failure behavior
|
## Failure behavior
|
||||||
|
|
||||||
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
|
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, invalid stored URLs, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
@@ -48,5 +50,6 @@ Before changing destination state behavior, inspect tests under `internal/state`
|
|||||||
- `.distributor.json` is the destination sentinel and state record.
|
- `.distributor.json` is the destination sentinel and state record.
|
||||||
- Embedded source manifests use the same validation rules as source bundles.
|
- Embedded source manifests use the same validation rules as source bundles.
|
||||||
- Generated outputs always record a transform id.
|
- Generated outputs always record a transform id.
|
||||||
|
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
|
||||||
- Comparison returns outcomes and reasons; it does not mutate storage.
|
- Comparison returns outcomes and reasons; it does not mutate storage.
|
||||||
- `distributor_version` is diagnostic metadata, not a comparison key.
|
- `distributor_version` is diagnostic metadata, not a comparison key.
|
||||||
|
|||||||
@@ -186,6 +186,9 @@ pipelines:
|
|||||||
path: /srv/www/reports/latest
|
path: /srv/www/reports/latest
|
||||||
path_mapping:
|
path_mapping:
|
||||||
mode: fixed
|
mode: fixed
|
||||||
|
links:
|
||||||
|
base_url: https://reports.example.com/latest
|
||||||
|
primary: auto
|
||||||
publish:
|
publish:
|
||||||
source: false
|
source: false
|
||||||
html: true
|
html: true
|
||||||
@@ -198,6 +201,32 @@ pipelines:
|
|||||||
|
|
||||||
The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
|
The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
|
||||||
|
|
||||||
|
## Static Site URLs
|
||||||
|
|
||||||
|
Use destination `links` when a destination backend root corresponds to a public HTTP or HTTPS URL:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
links:
|
||||||
|
base_url: https://reports.example.com/archive
|
||||||
|
primary: auto
|
||||||
|
```
|
||||||
|
|
||||||
|
Distributor records URLs in `.distributor.json`; it does not publish notifications or infer URLs from local, SSH, or S3 backend fields.
|
||||||
|
|
||||||
|
For archive-style destinations, URLs include the destination bundle path. A source bundle under `daily/brentwood/2026-06-01` with `base_url: https://reports.example.com/archive` can produce:
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://reports.example.com/archive/daily/brentwood/2026-06-01/report.html
|
||||||
|
```
|
||||||
|
|
||||||
|
For fixed destinations, URLs are rooted at `links.base_url`. A fixed HTML index destination with `base_url: https://reports.example.com/latest` records:
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://reports.example.com/latest/
|
||||||
|
```
|
||||||
|
|
||||||
|
`index.html` outputs use directory-style URLs. Other outputs include their filename. The primary URL is selected from the published outputs using the destination `links.primary` policy.
|
||||||
|
|
||||||
## Source Validation and Inspection
|
## Source Validation and Inspection
|
||||||
|
|
||||||
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
|
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
|
||||||
@@ -228,7 +257,7 @@ Dry-run output is useful before publishing to confirm actions such as `publish_n
|
|||||||
|
|
||||||
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
|
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
|
||||||
|
|
||||||
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, output records, final counters, and partial failure records. Fixed action records include `path_mapping: "fixed"`, and the summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
|
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, path mapping modes, optional link URLs, output records, final counters, and partial failure records. The summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
|
||||||
|
|
||||||
## Retry and Replacement Behavior
|
## Retry and Replacement Behavior
|
||||||
|
|
||||||
|
|||||||
@@ -17,6 +17,9 @@ pipelines:
|
|||||||
path: workspace/published/archive-and-latest/latest
|
path: workspace/published/archive-and-latest/latest
|
||||||
path_mapping:
|
path_mapping:
|
||||||
mode: fixed
|
mode: fixed
|
||||||
|
links:
|
||||||
|
base_url: https://reports.example.com/latest
|
||||||
|
primary: auto
|
||||||
publish:
|
publish:
|
||||||
source: false
|
source: false
|
||||||
html: true
|
html: true
|
||||||
|
|||||||
@@ -164,6 +164,7 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
|
|||||||
PathMapping: destination.PathMap.Mode,
|
PathMapping: destination.PathMap.Mode,
|
||||||
Publish: *destination.Publish,
|
Publish: *destination.Publish,
|
||||||
Transform: destination.Transform,
|
Transform: destination.Transform,
|
||||||
|
Links: destination.Links,
|
||||||
Transformers: transforms,
|
Transformers: transforms,
|
||||||
Transfer: destination.Transfer,
|
Transfer: destination.Transfer,
|
||||||
DistributorVersion: Version,
|
DistributorVersion: Version,
|
||||||
@@ -475,6 +476,7 @@ type runActionResult struct {
|
|||||||
DestinationPath string `json:"destination_path"`
|
DestinationPath string `json:"destination_path"`
|
||||||
PathMapping string `json:"path_mapping,omitempty"`
|
PathMapping string `json:"path_mapping,omitempty"`
|
||||||
Action string `json:"action"`
|
Action string `json:"action"`
|
||||||
|
PrimaryURL string `json:"primary_url,omitempty"`
|
||||||
Reason string `json:"reason,omitempty"`
|
Reason string `json:"reason,omitempty"`
|
||||||
Outputs []runOutputResult `json:"outputs"`
|
Outputs []runOutputResult `json:"outputs"`
|
||||||
}
|
}
|
||||||
@@ -484,6 +486,7 @@ type runOutputResult struct {
|
|||||||
Kind string `json:"kind"`
|
Kind string `json:"kind"`
|
||||||
SourcePath string `json:"source_path,omitempty"`
|
SourcePath string `json:"source_path,omitempty"`
|
||||||
Transform string `json:"transform,omitempty"`
|
Transform string `json:"transform,omitempty"`
|
||||||
|
URL string `json:"url,omitempty"`
|
||||||
SHA256 string `json:"sha256"`
|
SHA256 string `json:"sha256"`
|
||||||
Size int64 `json:"size"`
|
Size int64 `json:"size"`
|
||||||
}
|
}
|
||||||
@@ -503,6 +506,7 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActi
|
|||||||
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
||||||
PathMapping: plan.PathMapping,
|
PathMapping: plan.PathMapping,
|
||||||
Action: "error",
|
Action: "error",
|
||||||
|
PrimaryURL: plan.PrimaryURL,
|
||||||
Reason: planErr.Error(),
|
Reason: planErr.Error(),
|
||||||
Outputs: []runOutputResult{},
|
Outputs: []runOutputResult{},
|
||||||
}
|
}
|
||||||
@@ -516,6 +520,7 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActi
|
|||||||
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
||||||
PathMapping: plan.PathMapping,
|
PathMapping: plan.PathMapping,
|
||||||
Action: string(plan.Action),
|
Action: string(plan.Action),
|
||||||
|
PrimaryURL: plan.PrimaryURL,
|
||||||
Reason: plan.Reason,
|
Reason: plan.Reason,
|
||||||
Outputs: runOutputsFromPlan(plan.Outputs),
|
Outputs: runOutputsFromPlan(plan.Outputs),
|
||||||
}
|
}
|
||||||
@@ -542,6 +547,7 @@ func runOutputsFromPlan(outputs []publish.Output) []runOutputResult {
|
|||||||
Kind: output.Kind,
|
Kind: output.Kind,
|
||||||
SourcePath: output.SourcePath,
|
SourcePath: output.SourcePath,
|
||||||
Transform: output.Transform,
|
Transform: output.Transform,
|
||||||
|
URL: output.URL,
|
||||||
SHA256: output.SHA256,
|
SHA256: output.SHA256,
|
||||||
Size: output.Size,
|
Size: output.Size,
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -215,6 +215,9 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
|
|||||||
if got, want := len(destinationState.Outputs), 2; got != want {
|
if got, want := len(destinationState.Outputs), 2; got != want {
|
||||||
t.Fatalf("state output count = %d, want %d", got, want)
|
t.Fatalf("state output count = %d, want %d", got, want)
|
||||||
}
|
}
|
||||||
|
if destinationState.Links != nil || destinationState.Outputs[0].URL != "" {
|
||||||
|
t.Fatalf("state links = %#v output URL=%q, want absent", destinationState.Links, destinationState.Outputs[0].URL)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
|
func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
|
||||||
@@ -232,6 +235,54 @@ func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunRecordsLinksForNestedBundlePath(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
writeSourceBundle(t, sourceRoot, "daily/brentwood", testBundleOptions{})
|
||||||
|
|
||||||
|
err := Run(context.Background(), RunOptions{
|
||||||
|
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative, "https://reports.example.com/archive", config.LinkPrimaryAuto, true, false, ""),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
destinationState := readStateFile(t, filepath.Join(destinationRoot, "daily", "brentwood", storage.StateFileName))
|
||||||
|
if destinationState.Links == nil || destinationState.Links.PrimaryURL != "https://reports.example.com/archive/daily/brentwood/report.md" {
|
||||||
|
t.Fatalf("state links = %#v, want source primary URL", destinationState.Links)
|
||||||
|
}
|
||||||
|
outputs := outputsByPath(destinationState.Outputs)
|
||||||
|
if outputs["report.md"].URL != "https://reports.example.com/archive/daily/brentwood/report.md" {
|
||||||
|
t.Fatalf("report URL = %q", outputs["report.md"].URL)
|
||||||
|
}
|
||||||
|
if outputs["summary.txt"].URL != "https://reports.example.com/archive/daily/brentwood/summary.txt" {
|
||||||
|
t.Fatalf("summary URL = %q", outputs["summary.txt"].URL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunRecordsLinksForFixedIndexDestination(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
writeSourceBundle(t, sourceRoot, "older", testBundleOptions{ID: "reports.older", Created: testutil.DefaultCreated})
|
||||||
|
writeSourceBundle(t, sourceRoot, "newer", testBundleOptions{ID: "reports.newer", Created: testutil.DefaultCreated.Add(time.Hour)})
|
||||||
|
|
||||||
|
err := Run(context.Background(), RunOptions{
|
||||||
|
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingFixed, "https://reports.example.com/latest", config.LinkPrimaryAuto, false, true, config.TransformModeIndex),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||||
|
if destinationState.Links == nil || destinationState.Links.PrimaryURL != "https://reports.example.com/latest/" {
|
||||||
|
t.Fatalf("state links = %#v, want fixed index primary URL", destinationState.Links)
|
||||||
|
}
|
||||||
|
if got, want := len(destinationState.Outputs), 1; got != want {
|
||||||
|
t.Fatalf("state output count = %d, want %d", got, want)
|
||||||
|
}
|
||||||
|
if destinationState.Outputs[0].Path != "index.html" || destinationState.Outputs[0].URL != "https://reports.example.com/latest/" {
|
||||||
|
t.Fatalf("state output = %#v, want index URL", destinationState.Outputs[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunFixedPathPublishesNewestBundleAtDestinationRoot(t *testing.T) {
|
func TestRunFixedPathPublishesNewestBundleAtDestinationRoot(t *testing.T) {
|
||||||
sourceRoot := t.TempDir()
|
sourceRoot := t.TempDir()
|
||||||
destinationRoot := t.TempDir()
|
destinationRoot := t.TempDir()
|
||||||
@@ -1264,6 +1315,37 @@ pipelines:
|
|||||||
`)
|
`)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeLocalConfigWithLinks(t *testing.T, sourceRoot, destinationRoot, pathMapping, baseURL, primary string, publishSource, publishHTML bool, transformMode string) string {
|
||||||
|
t.Helper()
|
||||||
|
transformConfig := ""
|
||||||
|
if publishHTML {
|
||||||
|
transformConfig = `
|
||||||
|
transform:
|
||||||
|
markdown_to_html:
|
||||||
|
enabled: true
|
||||||
|
mode: ` + transformMode
|
||||||
|
}
|
||||||
|
return writeConfigFile(t, `
|
||||||
|
pipelines:
|
||||||
|
- id: reports
|
||||||
|
source:
|
||||||
|
backend: local
|
||||||
|
path: `+sourceRoot+`
|
||||||
|
destinations:
|
||||||
|
- id: archive
|
||||||
|
backend: local
|
||||||
|
path: `+destinationRoot+`
|
||||||
|
path_mapping:
|
||||||
|
mode: `+pathMapping+`
|
||||||
|
links:
|
||||||
|
base_url: `+baseURL+`
|
||||||
|
primary: `+primary+`
|
||||||
|
publish:
|
||||||
|
source: `+fmt.Sprintf("%t", publishSource)+`
|
||||||
|
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
|
||||||
|
`)
|
||||||
|
}
|
||||||
|
|
||||||
func writeLocalConfigWithMarkdownTransform(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool, mode, input string) string {
|
func writeLocalConfigWithMarkdownTransform(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool, mode, input string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
enabled := publishHTML
|
enabled := publishHTML
|
||||||
@@ -1344,6 +1426,14 @@ func readStateFile(t *testing.T, path string) state.DistributorState {
|
|||||||
return testutil.ReadDestinationState(t, path)
|
return testutil.ReadDestinationState(t, path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func outputsByPath(outputs []state.OutputFile) map[string]state.OutputFile {
|
||||||
|
byPath := make(map[string]state.OutputFile, len(outputs))
|
||||||
|
for _, output := range outputs {
|
||||||
|
byPath[output.Path] = output
|
||||||
|
}
|
||||||
|
return byPath
|
||||||
|
}
|
||||||
|
|
||||||
func assertFile(t *testing.T, path, want string) {
|
func assertFile(t *testing.T, path, want string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
|
|||||||
@@ -675,6 +675,58 @@ pipelines:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteRunJSONDryRunReportsLinks(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
|
||||||
|
configPath := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
if err := os.WriteFile(configPath, []byte(`
|
||||||
|
pipelines:
|
||||||
|
- id: reports
|
||||||
|
source:
|
||||||
|
backend: local
|
||||||
|
path: `+sourceRoot+`
|
||||||
|
destinations:
|
||||||
|
- id: web
|
||||||
|
backend: local
|
||||||
|
path: `+destinationRoot+`
|
||||||
|
links:
|
||||||
|
base_url: https://reports.example.com/archive
|
||||||
|
primary: source
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatalf("write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
code := Execute(context.Background(), []string{"run", "--config", configPath, "--dry-run", "--format", "json"}, &stdout, &stderr)
|
||||||
|
|
||||||
|
if code != exitOK {
|
||||||
|
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
|
||||||
|
}
|
||||||
|
envelope := decodeEnvelope(t, &stdout)
|
||||||
|
result := envelopeResult(t, envelope)
|
||||||
|
actions, ok := result["actions"].([]any)
|
||||||
|
if !ok || len(actions) != 1 {
|
||||||
|
t.Fatalf("actions = %#v, want one action", result["actions"])
|
||||||
|
}
|
||||||
|
action, ok := actions[0].(map[string]any)
|
||||||
|
if !ok || action["primary_url"] != "https://reports.example.com/archive/report.md" {
|
||||||
|
t.Fatalf("action = %#v, want primary URL", actions[0])
|
||||||
|
}
|
||||||
|
outputs, ok := action["outputs"].([]any)
|
||||||
|
if !ok || len(outputs) != 2 {
|
||||||
|
t.Fatalf("outputs = %#v, want two outputs", action["outputs"])
|
||||||
|
}
|
||||||
|
output, ok := outputs[0].(map[string]any)
|
||||||
|
if !ok || output["url"] != "https://reports.example.com/archive/report.md" {
|
||||||
|
t.Fatalf("output = %#v, want output URL", outputs[0])
|
||||||
|
}
|
||||||
|
if stderr.Len() != 0 {
|
||||||
|
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteRunJSONWarningsAreStructured(t *testing.T) {
|
func TestExecuteRunJSONWarningsAreStructured(t *testing.T) {
|
||||||
name := "DISTRIBUTOR_TEST_CLI_JSON_SECRET"
|
name := "DISTRIBUTOR_TEST_CLI_JSON_SECRET"
|
||||||
t.Setenv(name, "process-value")
|
t.Setenv(name, "process-value")
|
||||||
|
|||||||
@@ -33,6 +33,7 @@ type Destination struct {
|
|||||||
Publish *PublishPolicy `yaml:"publish"`
|
Publish *PublishPolicy `yaml:"publish"`
|
||||||
Transform Transform `yaml:"transform"`
|
Transform Transform `yaml:"transform"`
|
||||||
PathMap PathMapping `yaml:"path_mapping"`
|
PathMap PathMapping `yaml:"path_mapping"`
|
||||||
|
Links *Links `yaml:"links"`
|
||||||
Transfer TransferPolicy `yaml:"transfer"`
|
Transfer TransferPolicy `yaml:"transfer"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -85,6 +86,11 @@ type PathMapping struct {
|
|||||||
Mode string `yaml:"mode"`
|
Mode string `yaml:"mode"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type Links struct {
|
||||||
|
BaseURL string `yaml:"base_url"`
|
||||||
|
Primary string `yaml:"primary"`
|
||||||
|
}
|
||||||
|
|
||||||
type TransferPolicy struct {
|
type TransferPolicy struct {
|
||||||
OnDestinationSame string `yaml:"on_destination_same"`
|
OnDestinationSame string `yaml:"on_destination_same"`
|
||||||
OnDestinationOlder string `yaml:"on_destination_older"`
|
OnDestinationOlder string `yaml:"on_destination_older"`
|
||||||
|
|||||||
@@ -30,6 +30,12 @@ const (
|
|||||||
PathMappingFixed = "fixed"
|
PathMappingFixed = "fixed"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
LinkPrimaryAuto = "auto"
|
||||||
|
LinkPrimaryHTML = "html"
|
||||||
|
LinkPrimarySource = "source"
|
||||||
|
)
|
||||||
|
|
||||||
const DefaultS3Region = "us-east-1"
|
const DefaultS3Region = "us-east-1"
|
||||||
|
|
||||||
func ApplyDefaults(cfg *Config) {
|
func ApplyDefaults(cfg *Config) {
|
||||||
@@ -51,6 +57,9 @@ func ApplyDefaults(cfg *Config) {
|
|||||||
if destination.PathMap.Mode == "" {
|
if destination.PathMap.Mode == "" {
|
||||||
destination.PathMap.Mode = PathMappingPreserveRelative
|
destination.PathMap.Mode = PathMappingPreserveRelative
|
||||||
}
|
}
|
||||||
|
if destination.Links != nil && destination.Links.Primary == "" {
|
||||||
|
destination.Links.Primary = LinkPrimaryAuto
|
||||||
|
}
|
||||||
if destination.Transfer.OnDestinationSame == "" {
|
if destination.Transfer.OnDestinationSame == "" {
|
||||||
destination.Transfer.OnDestinationSame = TransferActionSkip
|
destination.Transfer.OnDestinationSame = TransferActionSkip
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -184,6 +184,30 @@ pipelines:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestLoadFileDefaultsLinksPrimaryToAuto(t *testing.T) {
|
||||||
|
cfg := loadConfig(t, `
|
||||||
|
pipelines:
|
||||||
|
- id: reports
|
||||||
|
source:
|
||||||
|
backend: local
|
||||||
|
path: /source
|
||||||
|
destinations:
|
||||||
|
- id: web
|
||||||
|
backend: local
|
||||||
|
path: /destination
|
||||||
|
links:
|
||||||
|
base_url: https://reports.example.com/archive
|
||||||
|
`)
|
||||||
|
|
||||||
|
links := cfg.Pipelines[0].Destinations[0].Links
|
||||||
|
if links == nil {
|
||||||
|
t.Fatal("links = nil, want config")
|
||||||
|
}
|
||||||
|
if links.BaseURL != "https://reports.example.com/archive" || links.Primary != LinkPrimaryAuto {
|
||||||
|
t.Fatalf("links = %#v, want base URL with auto primary", links)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLoadFileValidBackendConfigs(t *testing.T) {
|
func TestLoadFileValidBackendConfigs(t *testing.T) {
|
||||||
tests := map[string]string{
|
tests := map[string]string{
|
||||||
"local": `
|
"local": `
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/url"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
@@ -59,6 +60,7 @@ func Validate(cfg Config) error {
|
|||||||
errs = validateDestinationBackend(errs, destinationContext, destination)
|
errs = validateDestinationBackend(errs, destinationContext, destination)
|
||||||
errs = validatePublishTransformPolicy(errs, destinationContext, destination.Publish, destination.Transform)
|
errs = validatePublishTransformPolicy(errs, destinationContext, destination.Publish, destination.Transform)
|
||||||
errs = validatePathMapping(errs, destinationContext+".path_mapping", destination.PathMap)
|
errs = validatePathMapping(errs, destinationContext+".path_mapping", destination.PathMap)
|
||||||
|
errs = validateLinks(errs, destinationContext+".links", destination.Links)
|
||||||
errs = validateTransferPolicy(errs, destinationContext+".transfer", destination.Transfer)
|
errs = validateTransferPolicy(errs, destinationContext+".transfer", destination.Transfer)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -179,6 +181,43 @@ func validatePathMapping(errs ValidationErrors, context string, mapping PathMapp
|
|||||||
return errs
|
return errs
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateLinks(errs ValidationErrors, context string, links *Links) ValidationErrors {
|
||||||
|
if links == nil {
|
||||||
|
return errs
|
||||||
|
}
|
||||||
|
if links.BaseURL == "" {
|
||||||
|
errs = append(errs, context+".base_url is required")
|
||||||
|
} else if err := validateLinkBaseURL(links.BaseURL); err != nil {
|
||||||
|
errs = append(errs, context+".base_url "+err.Error())
|
||||||
|
}
|
||||||
|
switch links.Primary {
|
||||||
|
case LinkPrimaryAuto, LinkPrimaryHTML, LinkPrimarySource:
|
||||||
|
default:
|
||||||
|
errs = append(errs, context+".primary must be "+LinkPrimaryAuto+", "+LinkPrimaryHTML+", or "+LinkPrimarySource)
|
||||||
|
}
|
||||||
|
return errs
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateLinkBaseURL(value string) error {
|
||||||
|
parsed, err := url.Parse(value)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("must be a valid URL")
|
||||||
|
}
|
||||||
|
if parsed.Scheme != "http" && parsed.Scheme != "https" {
|
||||||
|
return fmt.Errorf("must use http or https")
|
||||||
|
}
|
||||||
|
if parsed.Host == "" {
|
||||||
|
return fmt.Errorf("must include a host")
|
||||||
|
}
|
||||||
|
if parsed.RawQuery != "" {
|
||||||
|
return fmt.Errorf("must not include a query string")
|
||||||
|
}
|
||||||
|
if parsed.Fragment != "" {
|
||||||
|
return fmt.Errorf("must not include a fragment")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateTransferPolicy(errs ValidationErrors, context string, policy TransferPolicy) ValidationErrors {
|
func validateTransferPolicy(errs ValidationErrors, context string, policy TransferPolicy) ValidationErrors {
|
||||||
if policy.OnDestinationSame != TransferActionSkip && policy.OnDestinationSame != TransferActionFail {
|
if policy.OnDestinationSame != TransferActionSkip && policy.OnDestinationSame != TransferActionFail {
|
||||||
errs = append(errs, context+".on_destination_same must be skip or fail")
|
errs = append(errs, context+".on_destination_same must be skip or fail")
|
||||||
|
|||||||
@@ -104,6 +104,47 @@ func TestValidatePathMapping(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestValidateLinks(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
links *Links
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{name: "absent links"},
|
||||||
|
{name: "http", links: &Links{BaseURL: "http://reports.example.com/archive", Primary: LinkPrimaryAuto}},
|
||||||
|
{name: "https", links: &Links{BaseURL: "https://reports.example.com/archive/", Primary: LinkPrimaryHTML}},
|
||||||
|
{name: "source primary", links: &Links{BaseURL: "https://reports.example.com", Primary: LinkPrimarySource}},
|
||||||
|
{name: "missing base", links: &Links{Primary: LinkPrimaryAuto}, wantErr: true},
|
||||||
|
{name: "ftp scheme", links: &Links{BaseURL: "ftp://reports.example.com", Primary: LinkPrimaryAuto}, wantErr: true},
|
||||||
|
{name: "missing host", links: &Links{BaseURL: "https:///archive", Primary: LinkPrimaryAuto}, wantErr: true},
|
||||||
|
{name: "query", links: &Links{BaseURL: "https://reports.example.com/archive?preview=1", Primary: LinkPrimaryAuto}, wantErr: true},
|
||||||
|
{name: "fragment", links: &Links{BaseURL: "https://reports.example.com/archive#top", Primary: LinkPrimaryAuto}, wantErr: true},
|
||||||
|
{name: "invalid primary", links: &Links{BaseURL: "https://reports.example.com", Primary: "document"}, wantErr: true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Config{Pipelines: []Pipeline{{
|
||||||
|
ID: "reports",
|
||||||
|
Source: Backend{Backend: BackendLocal, Path: "/source"},
|
||||||
|
Destinations: []Destination{{
|
||||||
|
ID: "web",
|
||||||
|
Backend: BackendLocal,
|
||||||
|
Path: "/destination",
|
||||||
|
Links: tt.links,
|
||||||
|
}},
|
||||||
|
}}}
|
||||||
|
ApplyDefaults(&cfg)
|
||||||
|
err := Validate(cfg)
|
||||||
|
if tt.wantErr && err == nil {
|
||||||
|
t.Fatal("Validate() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !tt.wantErr && err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
type publishTransformPolicyCase struct {
|
type publishTransformPolicyCase struct {
|
||||||
name string
|
name string
|
||||||
publish PublishPolicy
|
publish PublishPolicy
|
||||||
|
|||||||
@@ -78,6 +78,9 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
|
|||||||
Source: state.SourceState{Manifest: req.SourceBundle.Manifest},
|
Source: state.SourceState{Manifest: req.SourceBundle.Manifest},
|
||||||
Outputs: stateOutputs(plan.Outputs),
|
Outputs: stateOutputs(plan.Outputs),
|
||||||
}
|
}
|
||||||
|
if plan.PrimaryURL != "" {
|
||||||
|
destinationState.Links = &state.LinkState{PrimaryURL: plan.PrimaryURL}
|
||||||
|
}
|
||||||
if err := state.Validate(destinationState); err != nil {
|
if err := state.Validate(destinationState); err != nil {
|
||||||
cleanup()
|
cleanup()
|
||||||
return err
|
return err
|
||||||
|
|||||||
116
internal/publish/links.go
Normal file
116
internal/publish/links.go
Normal file
@@ -0,0 +1,116 @@
|
|||||||
|
package publish
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||||
|
)
|
||||||
|
|
||||||
|
func PlanLinks(req Request, outputs []Output) ([]Output, string, error) {
|
||||||
|
if req.Links == nil {
|
||||||
|
return outputs, "", nil
|
||||||
|
}
|
||||||
|
linked := make([]Output, 0, len(outputs))
|
||||||
|
for _, output := range outputs {
|
||||||
|
outputURL, err := OutputURL(req.Links.BaseURL, req.DestinationBundlePath, output.DestinationPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", err
|
||||||
|
}
|
||||||
|
output.URL = outputURL
|
||||||
|
linked = append(linked, output)
|
||||||
|
}
|
||||||
|
return linked, primaryURL(linked, req.Links.Primary), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func OutputURL(baseURL, destinationBundlePath, outputPath string) (string, error) {
|
||||||
|
parsed, err := url.Parse(baseURL)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("link base URL: %w", err)
|
||||||
|
}
|
||||||
|
segments := pathSegments(parsed.Path)
|
||||||
|
segments = append(segments, pathSegments(destinationBundlePath)...)
|
||||||
|
outputSegments := pathSegments(outputPath)
|
||||||
|
trailingSlash := false
|
||||||
|
if len(outputSegments) > 0 && outputSegments[len(outputSegments)-1] == "index.html" {
|
||||||
|
outputSegments = outputSegments[:len(outputSegments)-1]
|
||||||
|
trailingSlash = true
|
||||||
|
}
|
||||||
|
segments = append(segments, outputSegments...)
|
||||||
|
parsed.Path = urlPath(segments, trailingSlash)
|
||||||
|
parsed.RawPath = ""
|
||||||
|
return parsed.String(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func pathSegments(value string) []string {
|
||||||
|
trimmed := strings.Trim(value, "/")
|
||||||
|
if trimmed == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return strings.Split(trimmed, "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func urlPath(segments []string, trailingSlash bool) string {
|
||||||
|
if len(segments) == 0 {
|
||||||
|
return "/"
|
||||||
|
}
|
||||||
|
path := "/" + strings.Join(segments, "/")
|
||||||
|
if trailingSlash && !strings.HasSuffix(path, "/") {
|
||||||
|
path += "/"
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
func primaryURL(outputs []Output, policy string) string {
|
||||||
|
switch policy {
|
||||||
|
case config.LinkPrimaryHTML:
|
||||||
|
return firstGeneratedHTMLURL(outputs)
|
||||||
|
case config.LinkPrimarySource:
|
||||||
|
return firstSourceURL(outputs)
|
||||||
|
default:
|
||||||
|
if url := firstIndexURL(outputs); url != "" {
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
if url := firstGeneratedHTMLURL(outputs); url != "" {
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
return firstSourceURL(outputs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstIndexURL(outputs []Output) string {
|
||||||
|
for _, output := range outputs {
|
||||||
|
if isIndexOutput(output.DestinationPath) {
|
||||||
|
return output.URL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstGeneratedHTMLURL(outputs []Output) string {
|
||||||
|
for _, output := range outputs {
|
||||||
|
if output.Kind == state.OutputKindGenerated && isHTMLOutput(output.DestinationPath) {
|
||||||
|
return output.URL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstSourceURL(outputs []Output) string {
|
||||||
|
for _, output := range outputs {
|
||||||
|
if output.Kind == state.OutputKindSource {
|
||||||
|
return output.URL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func isIndexOutput(path string) bool {
|
||||||
|
return path == "index.html" || strings.HasSuffix(path, "/index.html")
|
||||||
|
}
|
||||||
|
|
||||||
|
func isHTMLOutput(path string) bool {
|
||||||
|
return strings.HasSuffix(path, ".html")
|
||||||
|
}
|
||||||
149
internal/publish/links_test.go
Normal file
149
internal/publish/links_test.go
Normal file
@@ -0,0 +1,149 @@
|
|||||||
|
package publish
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestOutputURLUsesURLPathSemantics(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
baseURL string
|
||||||
|
destinationBundlePath string
|
||||||
|
outputPath string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "nested non index",
|
||||||
|
baseURL: "https://reports.example.com/archive",
|
||||||
|
destinationBundlePath: "daily/brentwood",
|
||||||
|
outputPath: "report.html",
|
||||||
|
want: "https://reports.example.com/archive/daily/brentwood/report.html",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nested index",
|
||||||
|
baseURL: "https://reports.example.com/archive",
|
||||||
|
destinationBundlePath: "daily/brentwood",
|
||||||
|
outputPath: "index.html",
|
||||||
|
want: "https://reports.example.com/archive/daily/brentwood/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "fixed index",
|
||||||
|
baseURL: "https://reports.example.com/latest",
|
||||||
|
destinationBundlePath: "",
|
||||||
|
outputPath: "index.html",
|
||||||
|
want: "https://reports.example.com/latest/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "escaped segments",
|
||||||
|
baseURL: "https://reports.example.com/archive root",
|
||||||
|
destinationBundlePath: "daily reports",
|
||||||
|
outputPath: "morning report.html",
|
||||||
|
want: "https://reports.example.com/archive%20root/daily%20reports/morning%20report.html",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nested output index",
|
||||||
|
baseURL: "https://reports.example.com",
|
||||||
|
destinationBundlePath: "daily",
|
||||||
|
outputPath: "site/index.html",
|
||||||
|
want: "https://reports.example.com/daily/site/",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := OutputURL(tt.baseURL, tt.destinationBundlePath, tt.outputPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("OutputURL() error = %v", err)
|
||||||
|
}
|
||||||
|
if got != tt.want {
|
||||||
|
t.Fatalf("OutputURL() = %q, want %q", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPlanLinksSelectsPrimaryURL(t *testing.T) {
|
||||||
|
outputs := []Output{
|
||||||
|
{
|
||||||
|
DestinationPath: "report.md",
|
||||||
|
Kind: state.OutputKindSource,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
DestinationPath: "report.html",
|
||||||
|
Kind: state.OutputKindGenerated,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
DestinationPath: "index.html",
|
||||||
|
Kind: state.OutputKindGenerated,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
primary string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "auto prefers index", primary: config.LinkPrimaryAuto, want: "https://reports.example.com/daily/"},
|
||||||
|
{name: "html uses first generated html", primary: config.LinkPrimaryHTML, want: "https://reports.example.com/daily/report.html"},
|
||||||
|
{name: "source uses first source", primary: config.LinkPrimarySource, want: "https://reports.example.com/daily/report.md"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
linked, primaryURL, err := PlanLinks(Request{
|
||||||
|
DestinationBundlePath: "daily",
|
||||||
|
Links: &config.Links{
|
||||||
|
BaseURL: "https://reports.example.com",
|
||||||
|
Primary: tt.primary,
|
||||||
|
},
|
||||||
|
}, outputs)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PlanLinks() error = %v", err)
|
||||||
|
}
|
||||||
|
if primaryURL != tt.want {
|
||||||
|
t.Fatalf("primary URL = %q, want %q", primaryURL, tt.want)
|
||||||
|
}
|
||||||
|
for index, output := range linked {
|
||||||
|
if output.URL == "" {
|
||||||
|
t.Fatalf("linked output %d has empty URL", index)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPlanLinksReturnsNoPrimaryWhenPolicyHasNoMatch(t *testing.T) {
|
||||||
|
linked, primaryURL, err := PlanLinks(Request{
|
||||||
|
DestinationBundlePath: "daily",
|
||||||
|
Links: &config.Links{
|
||||||
|
BaseURL: "https://reports.example.com",
|
||||||
|
Primary: config.LinkPrimaryHTML,
|
||||||
|
},
|
||||||
|
}, []Output{{
|
||||||
|
DestinationPath: "report.md",
|
||||||
|
Kind: state.OutputKindSource,
|
||||||
|
}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PlanLinks() error = %v", err)
|
||||||
|
}
|
||||||
|
if primaryURL != "" {
|
||||||
|
t.Fatalf("primary URL = %q, want empty", primaryURL)
|
||||||
|
}
|
||||||
|
if linked[0].URL != "https://reports.example.com/daily/report.md" {
|
||||||
|
t.Fatalf("linked URL = %q", linked[0].URL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPlanLinksLeavesOutputsUnchangedWithoutConfig(t *testing.T) {
|
||||||
|
outputs := []Output{{DestinationPath: "report.md", Kind: state.OutputKindSource}}
|
||||||
|
linked, primaryURL, err := PlanLinks(Request{}, outputs)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PlanLinks() error = %v", err)
|
||||||
|
}
|
||||||
|
if primaryURL != "" {
|
||||||
|
t.Fatalf("primary URL = %q, want empty", primaryURL)
|
||||||
|
}
|
||||||
|
if linked[0].URL != "" {
|
||||||
|
t.Fatalf("output URL = %q, want empty", linked[0].URL)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -105,6 +105,7 @@ func stateOutputs(outputs []Output) []state.OutputFile {
|
|||||||
Kind: output.Kind,
|
Kind: output.Kind,
|
||||||
SourcePath: output.SourcePath,
|
SourcePath: output.SourcePath,
|
||||||
Transform: output.Transform,
|
Transform: output.Transform,
|
||||||
|
URL: output.URL,
|
||||||
SHA256: output.SHA256,
|
SHA256: output.SHA256,
|
||||||
Size: output.Size,
|
Size: output.Size,
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -33,6 +33,7 @@ type Request struct {
|
|||||||
PathMapping string
|
PathMapping string
|
||||||
Publish config.PublishPolicy
|
Publish config.PublishPolicy
|
||||||
Transform config.Transform
|
Transform config.Transform
|
||||||
|
Links *config.Links
|
||||||
Transformers TransformerResolver
|
Transformers TransformerResolver
|
||||||
Transfer config.TransferPolicy
|
Transfer config.TransferPolicy
|
||||||
DistributorVersion string
|
DistributorVersion string
|
||||||
@@ -53,6 +54,7 @@ type Plan struct {
|
|||||||
Action Action
|
Action Action
|
||||||
Reason string
|
Reason string
|
||||||
Force bool
|
Force bool
|
||||||
|
PrimaryURL string
|
||||||
Outputs []Output
|
Outputs []Output
|
||||||
ExistingState *state.DistributorState
|
ExistingState *state.DistributorState
|
||||||
}
|
}
|
||||||
@@ -62,6 +64,7 @@ type Output struct {
|
|||||||
DestinationPath string
|
DestinationPath string
|
||||||
Kind string
|
Kind string
|
||||||
Transform string
|
Transform string
|
||||||
|
URL string
|
||||||
Data []byte
|
Data []byte
|
||||||
SHA256 string
|
SHA256 string
|
||||||
Size int64
|
Size int64
|
||||||
@@ -75,6 +78,10 @@ func Build(ctx context.Context, req Request) (Plan, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return Plan{}, err
|
return Plan{}, err
|
||||||
}
|
}
|
||||||
|
outputs, primaryURL, err := PlanLinks(req, outputs)
|
||||||
|
if err != nil {
|
||||||
|
return Plan{}, err
|
||||||
|
}
|
||||||
status, err := inspectDestination(ctx, req.DestinationBackend, req.DestinationBundlePath)
|
status, err := inspectDestination(ctx, req.DestinationBackend, req.DestinationBundlePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Plan{}, err
|
return Plan{}, err
|
||||||
@@ -91,6 +98,7 @@ func Build(ctx context.Context, req Request) (Plan, error) {
|
|||||||
Action: action,
|
Action: action,
|
||||||
Reason: reason,
|
Reason: reason,
|
||||||
Force: action == ActionForceReplace,
|
Force: action == ActionForceReplace,
|
||||||
|
PrimaryURL: primaryURL,
|
||||||
Outputs: outputs,
|
Outputs: outputs,
|
||||||
ExistingState: status.State,
|
ExistingState: status.State,
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ type DistributorState struct {
|
|||||||
DestinationID string
|
DestinationID string
|
||||||
PublishedAt time.Time
|
PublishedAt time.Time
|
||||||
Source SourceState
|
Source SourceState
|
||||||
|
Links *LinkState
|
||||||
Outputs []OutputFile
|
Outputs []OutputFile
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -26,11 +27,16 @@ type SourceState struct {
|
|||||||
Manifest bundle.Manifest
|
Manifest bundle.Manifest
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type LinkState struct {
|
||||||
|
PrimaryURL string
|
||||||
|
}
|
||||||
|
|
||||||
type OutputFile struct {
|
type OutputFile struct {
|
||||||
Path string
|
Path string
|
||||||
Kind string
|
Kind string
|
||||||
SourcePath string
|
SourcePath string
|
||||||
Transform string
|
Transform string
|
||||||
|
URL string
|
||||||
SHA256 string
|
SHA256 string
|
||||||
Size int64
|
Size int64
|
||||||
}
|
}
|
||||||
@@ -42,6 +48,7 @@ type rawDistributorState struct {
|
|||||||
DestinationID *string `json:"destination_id"`
|
DestinationID *string `json:"destination_id"`
|
||||||
PublishedAt *string `json:"published_at"`
|
PublishedAt *string `json:"published_at"`
|
||||||
Source *rawSourceState `json:"source"`
|
Source *rawSourceState `json:"source"`
|
||||||
|
Links *rawLinkState `json:"links"`
|
||||||
Outputs []rawOutputFile `json:"outputs"`
|
Outputs []rawOutputFile `json:"outputs"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -49,11 +56,16 @@ type rawSourceState struct {
|
|||||||
Manifest json.RawMessage `json:"manifest"`
|
Manifest json.RawMessage `json:"manifest"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type rawLinkState struct {
|
||||||
|
PrimaryURL string `json:"primary_url"`
|
||||||
|
}
|
||||||
|
|
||||||
type rawOutputFile struct {
|
type rawOutputFile struct {
|
||||||
Path *string `json:"path"`
|
Path *string `json:"path"`
|
||||||
Kind *string `json:"kind"`
|
Kind *string `json:"kind"`
|
||||||
SourcePath *string `json:"source_path"`
|
SourcePath *string `json:"source_path"`
|
||||||
Transform string `json:"transform"`
|
Transform string `json:"transform"`
|
||||||
|
URL string `json:"url"`
|
||||||
SHA256 *string `json:"sha256"`
|
SHA256 *string `json:"sha256"`
|
||||||
Size *int64 `json:"size"`
|
Size *int64 `json:"size"`
|
||||||
}
|
}
|
||||||
@@ -112,6 +124,9 @@ func parseRaw(raw rawDistributorState) (DistributorState, error) {
|
|||||||
return DistributorState{}, fmt.Errorf("state source.manifest: %w", err)
|
return DistributorState{}, fmt.Errorf("state source.manifest: %w", err)
|
||||||
}
|
}
|
||||||
state.Source.Manifest = manifest
|
state.Source.Manifest = manifest
|
||||||
|
if raw.Links != nil {
|
||||||
|
state.Links = &LinkState{PrimaryURL: raw.Links.PrimaryURL}
|
||||||
|
}
|
||||||
if raw.Outputs == nil {
|
if raw.Outputs == nil {
|
||||||
return DistributorState{}, fmt.Errorf("state outputs is required")
|
return DistributorState{}, fmt.Errorf("state outputs is required")
|
||||||
}
|
}
|
||||||
@@ -161,6 +176,7 @@ func parseOutput(index int, raw rawOutputFile) (OutputFile, error) {
|
|||||||
Kind: *raw.Kind,
|
Kind: *raw.Kind,
|
||||||
SourcePath: *raw.SourcePath,
|
SourcePath: *raw.SourcePath,
|
||||||
Transform: raw.Transform,
|
Transform: raw.Transform,
|
||||||
|
URL: raw.URL,
|
||||||
SHA256: *raw.SHA256,
|
SHA256: *raw.SHA256,
|
||||||
Size: *raw.Size,
|
Size: *raw.Size,
|
||||||
}, nil
|
}, nil
|
||||||
@@ -181,6 +197,7 @@ func (s DistributorState) MarshalJSON() ([]byte, error) {
|
|||||||
DestinationID string `json:"destination_id"`
|
DestinationID string `json:"destination_id"`
|
||||||
PublishedAt string `json:"published_at"`
|
PublishedAt string `json:"published_at"`
|
||||||
Source sourceJSON `json:"source"`
|
Source sourceJSON `json:"source"`
|
||||||
|
Links *LinkState `json:"links,omitempty"`
|
||||||
Outputs []OutputFile `json:"outputs"`
|
Outputs []OutputFile `json:"outputs"`
|
||||||
}
|
}
|
||||||
return json.Marshal(stateJSON{
|
return json.Marshal(stateJSON{
|
||||||
@@ -190,16 +207,25 @@ func (s DistributorState) MarshalJSON() ([]byte, error) {
|
|||||||
DestinationID: s.DestinationID,
|
DestinationID: s.DestinationID,
|
||||||
PublishedAt: s.PublishedAtString(),
|
PublishedAt: s.PublishedAtString(),
|
||||||
Source: sourceJSON{Manifest: s.Source.Manifest},
|
Source: sourceJSON{Manifest: s.Source.Manifest},
|
||||||
|
Links: s.Links,
|
||||||
Outputs: s.Outputs,
|
Outputs: s.Outputs,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (l LinkState) MarshalJSON() ([]byte, error) {
|
||||||
|
type linkJSON struct {
|
||||||
|
PrimaryURL string `json:"primary_url,omitempty"`
|
||||||
|
}
|
||||||
|
return json.Marshal(linkJSON{PrimaryURL: l.PrimaryURL})
|
||||||
|
}
|
||||||
|
|
||||||
func (o OutputFile) MarshalJSON() ([]byte, error) {
|
func (o OutputFile) MarshalJSON() ([]byte, error) {
|
||||||
type outputJSON struct {
|
type outputJSON struct {
|
||||||
Path string `json:"path"`
|
Path string `json:"path"`
|
||||||
Kind string `json:"kind"`
|
Kind string `json:"kind"`
|
||||||
SourcePath string `json:"source_path"`
|
SourcePath string `json:"source_path"`
|
||||||
Transform string `json:"transform,omitempty"`
|
Transform string `json:"transform,omitempty"`
|
||||||
|
URL string `json:"url,omitempty"`
|
||||||
SHA256 string `json:"sha256"`
|
SHA256 string `json:"sha256"`
|
||||||
Size int64 `json:"size"`
|
Size int64 `json:"size"`
|
||||||
}
|
}
|
||||||
@@ -208,6 +234,7 @@ func (o OutputFile) MarshalJSON() ([]byte, error) {
|
|||||||
Kind: o.Kind,
|
Kind: o.Kind,
|
||||||
SourcePath: o.SourcePath,
|
SourcePath: o.SourcePath,
|
||||||
Transform: o.Transform,
|
Transform: o.Transform,
|
||||||
|
URL: o.URL,
|
||||||
SHA256: o.SHA256,
|
SHA256: o.SHA256,
|
||||||
Size: o.Size,
|
Size: o.Size,
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -29,6 +29,22 @@ func TestParseValidState(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestParseValidStateWithLinks(t *testing.T) {
|
||||||
|
body := strings.Replace(validStateJSON(t), `"outputs": [`, `"links": {"primary_url": "https://reports.example.com/archive/report.md"},`+"\n "+`"outputs": [`, 1)
|
||||||
|
body = strings.Replace(body, `"source_path": "report.md",`, `"source_path": "report.md",`+"\n "+`"url": "https://reports.example.com/archive/report.md",`, 1)
|
||||||
|
|
||||||
|
state, err := Parse([]byte(body))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if state.Links == nil || state.Links.PrimaryURL != "https://reports.example.com/archive/report.md" {
|
||||||
|
t.Fatalf("links = %#v, want primary URL", state.Links)
|
||||||
|
}
|
||||||
|
if state.Outputs[0].URL != "https://reports.example.com/archive/report.md" {
|
||||||
|
t.Fatalf("output URL = %q", state.Outputs[0].URL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseNormalizesPublishedAtOffset(t *testing.T) {
|
func TestParseNormalizesPublishedAtOffset(t *testing.T) {
|
||||||
body := strings.Replace(validStateJSON(t), `"published_at": "2026-05-30T11:12:00Z"`, `"published_at": "2026-05-30T13:12:00+02:00"`, 1)
|
body := strings.Replace(validStateJSON(t), `"published_at": "2026-05-30T11:12:00Z"`, `"published_at": "2026-05-30T13:12:00+02:00"`, 1)
|
||||||
state, err := Parse([]byte(body))
|
state, err := Parse([]byte(body))
|
||||||
@@ -145,6 +161,12 @@ func TestParseRejectsInvalidOutputMetadata(t *testing.T) {
|
|||||||
"negative size": func(s *DistributorState) {
|
"negative size": func(s *DistributorState) {
|
||||||
s.Outputs[0].Size = -1
|
s.Outputs[0].Size = -1
|
||||||
},
|
},
|
||||||
|
"invalid output url": func(s *DistributorState) {
|
||||||
|
s.Outputs[0].URL = "file:///tmp/report.md"
|
||||||
|
},
|
||||||
|
"invalid primary url": func(s *DistributorState) {
|
||||||
|
s.Links = &LinkState{PrimaryURL: "file:///tmp/report.md"}
|
||||||
|
},
|
||||||
}
|
}
|
||||||
for name, mutate := range tests {
|
for name, mutate := range tests {
|
||||||
t.Run(name, func(t *testing.T) {
|
t.Run(name, func(t *testing.T) {
|
||||||
@@ -188,6 +210,36 @@ func TestMarshalNormalizesPublishedAtUTC(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMarshalIncludesLinksWhenPresent(t *testing.T) {
|
||||||
|
source := validManifest(t)
|
||||||
|
state := DistributorState{
|
||||||
|
SchemaVersion: SchemaVersion,
|
||||||
|
PipelineID: "reports",
|
||||||
|
DestinationID: "archive",
|
||||||
|
PublishedAt: time.Date(2026, 5, 30, 11, 12, 0, 0, time.UTC),
|
||||||
|
Source: SourceState{Manifest: source},
|
||||||
|
Links: &LinkState{PrimaryURL: "https://reports.example.com/archive/report.md"},
|
||||||
|
Outputs: []OutputFile{{
|
||||||
|
Path: "report.md",
|
||||||
|
Kind: OutputKindSource,
|
||||||
|
SourcePath: "report.md",
|
||||||
|
URL: "https://reports.example.com/archive/report.md",
|
||||||
|
SHA256: source.Files[0].SHA256,
|
||||||
|
Size: source.Files[0].Size,
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
data, err := json.Marshal(state)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(data), `"links":{"primary_url":"https://reports.example.com/archive/report.md"}`) {
|
||||||
|
t.Fatalf("json = %s, want links primary URL", data)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(data), `"url":"https://reports.example.com/archive/report.md"`) {
|
||||||
|
t.Fatalf("json = %s, want output URL", data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func validStateJSON(t *testing.T) string {
|
func validStateJSON(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return validStateWithManifestJSON(t, manifestJSON(t))
|
return validStateWithManifestJSON(t, manifestJSON(t))
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package state
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/url"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||||
@@ -28,6 +29,11 @@ func Validate(s DistributorState) error {
|
|||||||
if err := validateEmbeddedManifest(s.Source.Manifest); err != nil {
|
if err := validateEmbeddedManifest(s.Source.Manifest); err != nil {
|
||||||
return fmt.Errorf("state source.manifest: %w", err)
|
return fmt.Errorf("state source.manifest: %w", err)
|
||||||
}
|
}
|
||||||
|
if s.Links != nil && s.Links.PrimaryURL != "" {
|
||||||
|
if err := validateStateURL(s.Links.PrimaryURL); err != nil {
|
||||||
|
return fmt.Errorf("state links.primary_url: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
if s.Outputs == nil {
|
if s.Outputs == nil {
|
||||||
return fmt.Errorf("state outputs is required")
|
return fmt.Errorf("state outputs is required")
|
||||||
}
|
}
|
||||||
@@ -63,6 +69,11 @@ func validateOutput(index int, output OutputFile) error {
|
|||||||
if output.Kind == OutputKindGenerated && output.Transform == "" {
|
if output.Kind == OutputKindGenerated && output.Transform == "" {
|
||||||
return fmt.Errorf("state outputs[%d].transform is required for generated output", index)
|
return fmt.Errorf("state outputs[%d].transform is required for generated output", index)
|
||||||
}
|
}
|
||||||
|
if output.URL != "" {
|
||||||
|
if err := validateStateURL(output.URL); err != nil {
|
||||||
|
return fmt.Errorf("state outputs[%d].url: %w", index, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
if err := bundle.ValidateDigest(output.SHA256); err != nil {
|
if err := bundle.ValidateDigest(output.SHA256); err != nil {
|
||||||
return fmt.Errorf("state outputs[%d].sha256: %w", index, err)
|
return fmt.Errorf("state outputs[%d].sha256: %w", index, err)
|
||||||
}
|
}
|
||||||
@@ -71,3 +82,23 @@ func validateOutput(index int, output OutputFile) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateStateURL(value string) error {
|
||||||
|
parsed, err := url.Parse(value)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if parsed.Scheme != "http" && parsed.Scheme != "https" {
|
||||||
|
return fmt.Errorf("must use http or https")
|
||||||
|
}
|
||||||
|
if parsed.Host == "" {
|
||||||
|
return fmt.Errorf("must include a host")
|
||||||
|
}
|
||||||
|
if parsed.RawQuery != "" {
|
||||||
|
return fmt.Errorf("must not include a query string")
|
||||||
|
}
|
||||||
|
if parsed.Fragment != "" {
|
||||||
|
return fmt.Errorf("must not include a fragment")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user