Add destination link metadata

This commit is contained in:
2026-06-01 21:42:00 +00:00
parent a8564035d3
commit 980ae15249
24 changed files with 737 additions and 16 deletions

View File

@@ -186,7 +186,7 @@ Command-specific JSON results:
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, optional fixed path mapping markers, output records, final counters, warnings, and partial failure records.
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, path mapping markers, optional primary URLs, output records with optional URLs, final counters, warnings, and partial failure records.
## Diagnostics

View File

@@ -117,6 +117,28 @@ Fixed destinations select the newest discovered source bundle by manifest `creat
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
## Destination Links
Destinations can record public URLs for published outputs:
```yaml
links:
base_url: https://reports.example.com/archive
primary: auto
```
`links.base_url` is an absolute `http` or `https` URL corresponding to the destination backend root. It may include a path prefix, but it must not include a query string or fragment. Distributor does not infer public URLs from backend config.
`links.primary` selects the top-level primary URL stored in destination state:
- `auto`: prefer `index.html`, then generated HTML, then source outputs.
- `html`: use the first generated HTML output.
- `source`: use the first copied source output.
If a destination has no `links` block, no URL metadata is generated. If a primary policy has no matching output, per-output URLs are still recorded and the top-level primary URL is omitted.
Output URLs are built from `links.base_url`, the destination bundle path, and the output path using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
## Reference
Top level:
@@ -156,6 +178,8 @@ Destination:
- `publish`: optional; defaults to source-only publication.
- `transform`: required only for generated HTML publication.
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
- `links.base_url`: optional links block; when present, `base_url` is required and must be an absolute HTTP or HTTPS URL without query string or fragment.
- `links.primary`: optional; defaults to `auto`. Accepted values are `auto`, `html`, and `source`.
- `transfer`: optional; defaults described below.
Accepted backend names:
@@ -246,6 +270,7 @@ Defaults are applied after YAML decoding and before validation:
- `publish.source: true`
- `publish.html: false`
- `path_mapping.mode: preserve_relative`
- `links.primary: auto` when a `links` block is present and `primary` is omitted
- `transfer.on_destination_same: skip`
- `transfer.on_destination_older: replace`
- `transfer.on_destination_newer: skip`

View File

@@ -6,9 +6,9 @@
## Inputs and outputs
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, writes summary output when stdout is supplied, and returns an aggregated error if any destination fails.
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, output format, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, writes text or JSON output when stdout is supplied, and returns an aggregated error if any destination fails.
`Validate` and `Inspect` accept a local path. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
`Validate` and `Inspect` accept either a local path or one configured pipeline source. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
## Run flow
@@ -17,11 +17,12 @@ The runner:
1. loads config from the supplied path or `config.DefaultConfigPath`;
2. opens the configured source backend;
3. discovers validated bundles from the source root;
4. opens each destination backend independently;
5. builds a publish plan for each bundle and destination;
6. prints plan lines and records summary counters;
7. executes publish or replacement plans unless dry-run is enabled;
8. invokes the notifier after successful publish or replacement actions.
4. selects source bundles for each destination according to destination path mapping;
5. opens each destination backend independently;
6. builds publish plans for the selected bundle and destination combinations;
7. prints plan lines or JSON action records and records summary counters;
8. executes publish or replacement plans unless dry-run is enabled;
9. invokes the notifier after successful publish or replacement actions.
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out.
@@ -47,7 +48,7 @@ Stdout write errors are returned immediately because the caller's requested outp
`internal/app` coordinates packages but does not own manifest validation rules, destination state comparison, storage path rules, output planning, transform rendering, or backend-specific filesystem behavior.
`Validate` and `Inspect` are local path commands. Remote execution wiring currently belongs to `Run`.
Configured-source `Validate` and `Inspect` share source backend construction with `Run` and do not open destinations.
## Tests

View File

@@ -26,6 +26,7 @@ Defaults are applied in `ApplyDefaults`:
- destination publish policy defaults to source output only;
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
- destination path mapping defaults to `preserve_relative`;
- destination link primary policy defaults to `auto` when a `links` block is present;
- `transfer.on_destination_same` defaults to `skip`;
- `transfer.on_destination_older` defaults to `replace`;
- `transfer.on_destination_newer` defaults to `skip`;
@@ -33,7 +34,7 @@ Defaults are applied in `ApplyDefaults`:
## Validation responsibilities
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, and valid transfer actions.
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
@@ -41,6 +42,8 @@ Transfer validation accepts `replace` for `on_destination_newer` and `on_conflic
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
Destination links are optional. When a `links` block is present, `base_url` is required, must use `http` or `https`, and must not include a query string or fragment. `primary` accepts `auto`, `html`, and `source`.
## Executable support boundary
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.

View File

@@ -6,9 +6,9 @@
## Inputs and outputs
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
Output is a plan with an action, reason, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
Output is a plan with an action, reason, optional primary URL, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
## Actions
@@ -28,6 +28,8 @@ The package uses `internal/state` for destination comparison, `internal/storage`
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
When link config is present, publish planning builds per-output URLs from `links.base_url`, the destination bundle path, and each output path. `index.html` outputs use directory-style URLs. The primary URL is selected from planned outputs according to the destination primary policy.
## Safety
Normal replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path before writing outputs and state. Failed writes trigger cleanup of outputs written during the failed attempt where practical.
@@ -40,6 +42,7 @@ Before changing publish behavior, inspect tests under `internal/publish` and run
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
- URL generation uses URL path semantics and does not infer public URLs from backend configuration.
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
- Forced replacement is explicit per run and deletes only within the destination bundle path.
- Publish execution writes destination state after selected outputs are written.

View File

@@ -19,13 +19,15 @@ Input is JSON destination state plus the current source manifest, pipeline id, d
- `source.manifest`
- `outputs`
`distributor_version` is optional diagnostic metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
`distributor_version` is optional diagnostic metadata. `links` is optional URL metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
The embedded `source.manifest` is validated with the same source manifest rules used by `internal/bundle`.
## Outputs
Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`.
Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`. Outputs may record `url` when the destination has link generation configured.
The optional top-level `links.primary_url` records the selected primary URL for the published destination bundle. It is omitted when link generation is not configured or when the destination primary policy has no matching output.
## Comparison
@@ -33,7 +35,7 @@ Comparison outcomes cover absent destination state, unmanaged destination conten
## Failure behavior
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, invalid stored URLs, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
## Boundaries
@@ -48,5 +50,6 @@ Before changing destination state behavior, inspect tests under `internal/state`
- `.distributor.json` is the destination sentinel and state record.
- Embedded source manifests use the same validation rules as source bundles.
- Generated outputs always record a transform id.
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
- Comparison returns outcomes and reasons; it does not mutate storage.
- `distributor_version` is diagnostic metadata, not a comparison key.

View File

@@ -186,6 +186,9 @@ pipelines:
path: /srv/www/reports/latest
path_mapping:
mode: fixed
links:
base_url: https://reports.example.com/latest
primary: auto
publish:
source: false
html: true
@@ -198,6 +201,32 @@ pipelines:
The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
## Static Site URLs
Use destination `links` when a destination backend root corresponds to a public HTTP or HTTPS URL:
```yaml
links:
base_url: https://reports.example.com/archive
primary: auto
```
Distributor records URLs in `.distributor.json`; it does not publish notifications or infer URLs from local, SSH, or S3 backend fields.
For archive-style destinations, URLs include the destination bundle path. A source bundle under `daily/brentwood/2026-06-01` with `base_url: https://reports.example.com/archive` can produce:
```text
https://reports.example.com/archive/daily/brentwood/2026-06-01/report.html
```
For fixed destinations, URLs are rooted at `links.base_url`. A fixed HTML index destination with `base_url: https://reports.example.com/latest` records:
```text
https://reports.example.com/latest/
```
`index.html` outputs use directory-style URLs. Other outputs include their filename. The primary URL is selected from the published outputs using the destination `links.primary` policy.
## Source Validation and Inspection
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
@@ -228,7 +257,7 @@ Dry-run output is useful before publishing to confirm actions such as `publish_n
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, output records, final counters, and partial failure records. Fixed action records include `path_mapping: "fixed"`, and the summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, path mapping modes, optional link URLs, output records, final counters, and partial failure records. The summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
## Retry and Replacement Behavior

View File

@@ -17,6 +17,9 @@ pipelines:
path: workspace/published/archive-and-latest/latest
path_mapping:
mode: fixed
links:
base_url: https://reports.example.com/latest
primary: auto
publish:
source: false
html: true

View File

@@ -164,6 +164,7 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
PathMapping: destination.PathMap.Mode,
Publish: *destination.Publish,
Transform: destination.Transform,
Links: destination.Links,
Transformers: transforms,
Transfer: destination.Transfer,
DistributorVersion: Version,
@@ -475,6 +476,7 @@ type runActionResult struct {
DestinationPath string `json:"destination_path"`
PathMapping string `json:"path_mapping,omitempty"`
Action string `json:"action"`
PrimaryURL string `json:"primary_url,omitempty"`
Reason string `json:"reason,omitempty"`
Outputs []runOutputResult `json:"outputs"`
}
@@ -484,6 +486,7 @@ type runOutputResult struct {
Kind string `json:"kind"`
SourcePath string `json:"source_path,omitempty"`
Transform string `json:"transform,omitempty"`
URL string `json:"url,omitempty"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
@@ -503,6 +506,7 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActi
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: "error",
PrimaryURL: plan.PrimaryURL,
Reason: planErr.Error(),
Outputs: []runOutputResult{},
}
@@ -516,6 +520,7 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActi
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: string(plan.Action),
PrimaryURL: plan.PrimaryURL,
Reason: plan.Reason,
Outputs: runOutputsFromPlan(plan.Outputs),
}
@@ -542,6 +547,7 @@ func runOutputsFromPlan(outputs []publish.Output) []runOutputResult {
Kind: output.Kind,
SourcePath: output.SourcePath,
Transform: output.Transform,
URL: output.URL,
SHA256: output.SHA256,
Size: output.Size,
})

View File

@@ -215,6 +215,9 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
if got, want := len(destinationState.Outputs), 2; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
}
if destinationState.Links != nil || destinationState.Outputs[0].URL != "" {
t.Fatalf("state links = %#v output URL=%q, want absent", destinationState.Links, destinationState.Outputs[0].URL)
}
}
func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
@@ -232,6 +235,54 @@ func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
}
}
func TestRunRecordsLinksForNestedBundlePath(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "daily/brentwood", testBundleOptions{})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative, "https://reports.example.com/archive", config.LinkPrimaryAuto, true, false, ""),
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, "daily", "brentwood", storage.StateFileName))
if destinationState.Links == nil || destinationState.Links.PrimaryURL != "https://reports.example.com/archive/daily/brentwood/report.md" {
t.Fatalf("state links = %#v, want source primary URL", destinationState.Links)
}
outputs := outputsByPath(destinationState.Outputs)
if outputs["report.md"].URL != "https://reports.example.com/archive/daily/brentwood/report.md" {
t.Fatalf("report URL = %q", outputs["report.md"].URL)
}
if outputs["summary.txt"].URL != "https://reports.example.com/archive/daily/brentwood/summary.txt" {
t.Fatalf("summary URL = %q", outputs["summary.txt"].URL)
}
}
func TestRunRecordsLinksForFixedIndexDestination(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "older", testBundleOptions{ID: "reports.older", Created: testutil.DefaultCreated})
writeSourceBundle(t, sourceRoot, "newer", testBundleOptions{ID: "reports.newer", Created: testutil.DefaultCreated.Add(time.Hour)})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingFixed, "https://reports.example.com/latest", config.LinkPrimaryAuto, false, true, config.TransformModeIndex),
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if destinationState.Links == nil || destinationState.Links.PrimaryURL != "https://reports.example.com/latest/" {
t.Fatalf("state links = %#v, want fixed index primary URL", destinationState.Links)
}
if got, want := len(destinationState.Outputs), 1; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
}
if destinationState.Outputs[0].Path != "index.html" || destinationState.Outputs[0].URL != "https://reports.example.com/latest/" {
t.Fatalf("state output = %#v, want index URL", destinationState.Outputs[0])
}
}
func TestRunFixedPathPublishesNewestBundleAtDestinationRoot(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
@@ -1264,6 +1315,37 @@ pipelines:
`)
}
func writeLocalConfigWithLinks(t *testing.T, sourceRoot, destinationRoot, pathMapping, baseURL, primary string, publishSource, publishHTML bool, transformMode string) string {
t.Helper()
transformConfig := ""
if publishHTML {
transformConfig = `
transform:
markdown_to_html:
enabled: true
mode: ` + transformMode
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
path_mapping:
mode: `+pathMapping+`
links:
base_url: `+baseURL+`
primary: `+primary+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
`)
}
func writeLocalConfigWithMarkdownTransform(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool, mode, input string) string {
t.Helper()
enabled := publishHTML
@@ -1344,6 +1426,14 @@ func readStateFile(t *testing.T, path string) state.DistributorState {
return testutil.ReadDestinationState(t, path)
}
func outputsByPath(outputs []state.OutputFile) map[string]state.OutputFile {
byPath := make(map[string]state.OutputFile, len(outputs))
for _, output := range outputs {
byPath[output.Path] = output
}
return byPath
}
func assertFile(t *testing.T, path, want string) {
t.Helper()
data, err := os.ReadFile(path)

View File

@@ -675,6 +675,58 @@ pipelines:
}
}
func TestExecuteRunJSONDryRunReportsLinks(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(`
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: web
backend: local
path: `+destinationRoot+`
links:
base_url: https://reports.example.com/archive
primary: source
`), 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"run", "--config", configPath, "--dry-run", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
result := envelopeResult(t, envelope)
actions, ok := result["actions"].([]any)
if !ok || len(actions) != 1 {
t.Fatalf("actions = %#v, want one action", result["actions"])
}
action, ok := actions[0].(map[string]any)
if !ok || action["primary_url"] != "https://reports.example.com/archive/report.md" {
t.Fatalf("action = %#v, want primary URL", actions[0])
}
outputs, ok := action["outputs"].([]any)
if !ok || len(outputs) != 2 {
t.Fatalf("outputs = %#v, want two outputs", action["outputs"])
}
output, ok := outputs[0].(map[string]any)
if !ok || output["url"] != "https://reports.example.com/archive/report.md" {
t.Fatalf("output = %#v, want output URL", outputs[0])
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestExecuteRunJSONWarningsAreStructured(t *testing.T) {
name := "DISTRIBUTOR_TEST_CLI_JSON_SECRET"
t.Setenv(name, "process-value")

View File

@@ -33,6 +33,7 @@ type Destination struct {
Publish *PublishPolicy `yaml:"publish"`
Transform Transform `yaml:"transform"`
PathMap PathMapping `yaml:"path_mapping"`
Links *Links `yaml:"links"`
Transfer TransferPolicy `yaml:"transfer"`
}
@@ -85,6 +86,11 @@ type PathMapping struct {
Mode string `yaml:"mode"`
}
type Links struct {
BaseURL string `yaml:"base_url"`
Primary string `yaml:"primary"`
}
type TransferPolicy struct {
OnDestinationSame string `yaml:"on_destination_same"`
OnDestinationOlder string `yaml:"on_destination_older"`

View File

@@ -30,6 +30,12 @@ const (
PathMappingFixed = "fixed"
)
const (
LinkPrimaryAuto = "auto"
LinkPrimaryHTML = "html"
LinkPrimarySource = "source"
)
const DefaultS3Region = "us-east-1"
func ApplyDefaults(cfg *Config) {
@@ -51,6 +57,9 @@ func ApplyDefaults(cfg *Config) {
if destination.PathMap.Mode == "" {
destination.PathMap.Mode = PathMappingPreserveRelative
}
if destination.Links != nil && destination.Links.Primary == "" {
destination.Links.Primary = LinkPrimaryAuto
}
if destination.Transfer.OnDestinationSame == "" {
destination.Transfer.OnDestinationSame = TransferActionSkip
}

View File

@@ -184,6 +184,30 @@ pipelines:
}
}
func TestLoadFileDefaultsLinksPrimaryToAuto(t *testing.T) {
cfg := loadConfig(t, `
pipelines:
- id: reports
source:
backend: local
path: /source
destinations:
- id: web
backend: local
path: /destination
links:
base_url: https://reports.example.com/archive
`)
links := cfg.Pipelines[0].Destinations[0].Links
if links == nil {
t.Fatal("links = nil, want config")
}
if links.BaseURL != "https://reports.example.com/archive" || links.Primary != LinkPrimaryAuto {
t.Fatalf("links = %#v, want base URL with auto primary", links)
}
}
func TestLoadFileValidBackendConfigs(t *testing.T) {
tests := map[string]string{
"local": `

View File

@@ -2,6 +2,7 @@ package config
import (
"fmt"
"net/url"
"regexp"
"strings"
)
@@ -59,6 +60,7 @@ func Validate(cfg Config) error {
errs = validateDestinationBackend(errs, destinationContext, destination)
errs = validatePublishTransformPolicy(errs, destinationContext, destination.Publish, destination.Transform)
errs = validatePathMapping(errs, destinationContext+".path_mapping", destination.PathMap)
errs = validateLinks(errs, destinationContext+".links", destination.Links)
errs = validateTransferPolicy(errs, destinationContext+".transfer", destination.Transfer)
}
}
@@ -179,6 +181,43 @@ func validatePathMapping(errs ValidationErrors, context string, mapping PathMapp
return errs
}
func validateLinks(errs ValidationErrors, context string, links *Links) ValidationErrors {
if links == nil {
return errs
}
if links.BaseURL == "" {
errs = append(errs, context+".base_url is required")
} else if err := validateLinkBaseURL(links.BaseURL); err != nil {
errs = append(errs, context+".base_url "+err.Error())
}
switch links.Primary {
case LinkPrimaryAuto, LinkPrimaryHTML, LinkPrimarySource:
default:
errs = append(errs, context+".primary must be "+LinkPrimaryAuto+", "+LinkPrimaryHTML+", or "+LinkPrimarySource)
}
return errs
}
func validateLinkBaseURL(value string) error {
parsed, err := url.Parse(value)
if err != nil {
return fmt.Errorf("must be a valid URL")
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("must use http or https")
}
if parsed.Host == "" {
return fmt.Errorf("must include a host")
}
if parsed.RawQuery != "" {
return fmt.Errorf("must not include a query string")
}
if parsed.Fragment != "" {
return fmt.Errorf("must not include a fragment")
}
return nil
}
func validateTransferPolicy(errs ValidationErrors, context string, policy TransferPolicy) ValidationErrors {
if policy.OnDestinationSame != TransferActionSkip && policy.OnDestinationSame != TransferActionFail {
errs = append(errs, context+".on_destination_same must be skip or fail")

View File

@@ -104,6 +104,47 @@ func TestValidatePathMapping(t *testing.T) {
}
}
func TestValidateLinks(t *testing.T) {
tests := []struct {
name string
links *Links
wantErr bool
}{
{name: "absent links"},
{name: "http", links: &Links{BaseURL: "http://reports.example.com/archive", Primary: LinkPrimaryAuto}},
{name: "https", links: &Links{BaseURL: "https://reports.example.com/archive/", Primary: LinkPrimaryHTML}},
{name: "source primary", links: &Links{BaseURL: "https://reports.example.com", Primary: LinkPrimarySource}},
{name: "missing base", links: &Links{Primary: LinkPrimaryAuto}, wantErr: true},
{name: "ftp scheme", links: &Links{BaseURL: "ftp://reports.example.com", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "missing host", links: &Links{BaseURL: "https:///archive", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "query", links: &Links{BaseURL: "https://reports.example.com/archive?preview=1", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "fragment", links: &Links{BaseURL: "https://reports.example.com/archive#top", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "invalid primary", links: &Links{BaseURL: "https://reports.example.com", Primary: "document"}, wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Config{Pipelines: []Pipeline{{
ID: "reports",
Source: Backend{Backend: BackendLocal, Path: "/source"},
Destinations: []Destination{{
ID: "web",
Backend: BackendLocal,
Path: "/destination",
Links: tt.links,
}},
}}}
ApplyDefaults(&cfg)
err := Validate(cfg)
if tt.wantErr && err == nil {
t.Fatal("Validate() error = nil, want error")
}
if !tt.wantErr && err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
type publishTransformPolicyCase struct {
name string
publish PublishPolicy

View File

@@ -78,6 +78,9 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
Source: state.SourceState{Manifest: req.SourceBundle.Manifest},
Outputs: stateOutputs(plan.Outputs),
}
if plan.PrimaryURL != "" {
destinationState.Links = &state.LinkState{PrimaryURL: plan.PrimaryURL}
}
if err := state.Validate(destinationState); err != nil {
cleanup()
return err

116
internal/publish/links.go Normal file
View File

@@ -0,0 +1,116 @@
package publish
import (
"fmt"
"net/url"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/state"
)
func PlanLinks(req Request, outputs []Output) ([]Output, string, error) {
if req.Links == nil {
return outputs, "", nil
}
linked := make([]Output, 0, len(outputs))
for _, output := range outputs {
outputURL, err := OutputURL(req.Links.BaseURL, req.DestinationBundlePath, output.DestinationPath)
if err != nil {
return nil, "", err
}
output.URL = outputURL
linked = append(linked, output)
}
return linked, primaryURL(linked, req.Links.Primary), nil
}
func OutputURL(baseURL, destinationBundlePath, outputPath string) (string, error) {
parsed, err := url.Parse(baseURL)
if err != nil {
return "", fmt.Errorf("link base URL: %w", err)
}
segments := pathSegments(parsed.Path)
segments = append(segments, pathSegments(destinationBundlePath)...)
outputSegments := pathSegments(outputPath)
trailingSlash := false
if len(outputSegments) > 0 && outputSegments[len(outputSegments)-1] == "index.html" {
outputSegments = outputSegments[:len(outputSegments)-1]
trailingSlash = true
}
segments = append(segments, outputSegments...)
parsed.Path = urlPath(segments, trailingSlash)
parsed.RawPath = ""
return parsed.String(), nil
}
func pathSegments(value string) []string {
trimmed := strings.Trim(value, "/")
if trimmed == "" {
return nil
}
return strings.Split(trimmed, "/")
}
func urlPath(segments []string, trailingSlash bool) string {
if len(segments) == 0 {
return "/"
}
path := "/" + strings.Join(segments, "/")
if trailingSlash && !strings.HasSuffix(path, "/") {
path += "/"
}
return path
}
func primaryURL(outputs []Output, policy string) string {
switch policy {
case config.LinkPrimaryHTML:
return firstGeneratedHTMLURL(outputs)
case config.LinkPrimarySource:
return firstSourceURL(outputs)
default:
if url := firstIndexURL(outputs); url != "" {
return url
}
if url := firstGeneratedHTMLURL(outputs); url != "" {
return url
}
return firstSourceURL(outputs)
}
}
func firstIndexURL(outputs []Output) string {
for _, output := range outputs {
if isIndexOutput(output.DestinationPath) {
return output.URL
}
}
return ""
}
func firstGeneratedHTMLURL(outputs []Output) string {
for _, output := range outputs {
if output.Kind == state.OutputKindGenerated && isHTMLOutput(output.DestinationPath) {
return output.URL
}
}
return ""
}
func firstSourceURL(outputs []Output) string {
for _, output := range outputs {
if output.Kind == state.OutputKindSource {
return output.URL
}
}
return ""
}
func isIndexOutput(path string) bool {
return path == "index.html" || strings.HasSuffix(path, "/index.html")
}
func isHTMLOutput(path string) bool {
return strings.HasSuffix(path, ".html")
}

View File

@@ -0,0 +1,149 @@
package publish
import (
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/state"
)
func TestOutputURLUsesURLPathSemantics(t *testing.T) {
tests := []struct {
name string
baseURL string
destinationBundlePath string
outputPath string
want string
}{
{
name: "nested non index",
baseURL: "https://reports.example.com/archive",
destinationBundlePath: "daily/brentwood",
outputPath: "report.html",
want: "https://reports.example.com/archive/daily/brentwood/report.html",
},
{
name: "nested index",
baseURL: "https://reports.example.com/archive",
destinationBundlePath: "daily/brentwood",
outputPath: "index.html",
want: "https://reports.example.com/archive/daily/brentwood/",
},
{
name: "fixed index",
baseURL: "https://reports.example.com/latest",
destinationBundlePath: "",
outputPath: "index.html",
want: "https://reports.example.com/latest/",
},
{
name: "escaped segments",
baseURL: "https://reports.example.com/archive root",
destinationBundlePath: "daily reports",
outputPath: "morning report.html",
want: "https://reports.example.com/archive%20root/daily%20reports/morning%20report.html",
},
{
name: "nested output index",
baseURL: "https://reports.example.com",
destinationBundlePath: "daily",
outputPath: "site/index.html",
want: "https://reports.example.com/daily/site/",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := OutputURL(tt.baseURL, tt.destinationBundlePath, tt.outputPath)
if err != nil {
t.Fatalf("OutputURL() error = %v", err)
}
if got != tt.want {
t.Fatalf("OutputURL() = %q, want %q", got, tt.want)
}
})
}
}
func TestPlanLinksSelectsPrimaryURL(t *testing.T) {
outputs := []Output{
{
DestinationPath: "report.md",
Kind: state.OutputKindSource,
},
{
DestinationPath: "report.html",
Kind: state.OutputKindGenerated,
},
{
DestinationPath: "index.html",
Kind: state.OutputKindGenerated,
},
}
tests := []struct {
name string
primary string
want string
}{
{name: "auto prefers index", primary: config.LinkPrimaryAuto, want: "https://reports.example.com/daily/"},
{name: "html uses first generated html", primary: config.LinkPrimaryHTML, want: "https://reports.example.com/daily/report.html"},
{name: "source uses first source", primary: config.LinkPrimarySource, want: "https://reports.example.com/daily/report.md"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
linked, primaryURL, err := PlanLinks(Request{
DestinationBundlePath: "daily",
Links: &config.Links{
BaseURL: "https://reports.example.com",
Primary: tt.primary,
},
}, outputs)
if err != nil {
t.Fatalf("PlanLinks() error = %v", err)
}
if primaryURL != tt.want {
t.Fatalf("primary URL = %q, want %q", primaryURL, tt.want)
}
for index, output := range linked {
if output.URL == "" {
t.Fatalf("linked output %d has empty URL", index)
}
}
})
}
}
func TestPlanLinksReturnsNoPrimaryWhenPolicyHasNoMatch(t *testing.T) {
linked, primaryURL, err := PlanLinks(Request{
DestinationBundlePath: "daily",
Links: &config.Links{
BaseURL: "https://reports.example.com",
Primary: config.LinkPrimaryHTML,
},
}, []Output{{
DestinationPath: "report.md",
Kind: state.OutputKindSource,
}})
if err != nil {
t.Fatalf("PlanLinks() error = %v", err)
}
if primaryURL != "" {
t.Fatalf("primary URL = %q, want empty", primaryURL)
}
if linked[0].URL != "https://reports.example.com/daily/report.md" {
t.Fatalf("linked URL = %q", linked[0].URL)
}
}
func TestPlanLinksLeavesOutputsUnchangedWithoutConfig(t *testing.T) {
outputs := []Output{{DestinationPath: "report.md", Kind: state.OutputKindSource}}
linked, primaryURL, err := PlanLinks(Request{}, outputs)
if err != nil {
t.Fatalf("PlanLinks() error = %v", err)
}
if primaryURL != "" {
t.Fatalf("primary URL = %q, want empty", primaryURL)
}
if linked[0].URL != "" {
t.Fatalf("output URL = %q, want empty", linked[0].URL)
}
}

View File

@@ -105,6 +105,7 @@ func stateOutputs(outputs []Output) []state.OutputFile {
Kind: output.Kind,
SourcePath: output.SourcePath,
Transform: output.Transform,
URL: output.URL,
SHA256: output.SHA256,
Size: output.Size,
})

View File

@@ -33,6 +33,7 @@ type Request struct {
PathMapping string
Publish config.PublishPolicy
Transform config.Transform
Links *config.Links
Transformers TransformerResolver
Transfer config.TransferPolicy
DistributorVersion string
@@ -53,6 +54,7 @@ type Plan struct {
Action Action
Reason string
Force bool
PrimaryURL string
Outputs []Output
ExistingState *state.DistributorState
}
@@ -62,6 +64,7 @@ type Output struct {
DestinationPath string
Kind string
Transform string
URL string
Data []byte
SHA256 string
Size int64
@@ -75,6 +78,10 @@ func Build(ctx context.Context, req Request) (Plan, error) {
if err != nil {
return Plan{}, err
}
outputs, primaryURL, err := PlanLinks(req, outputs)
if err != nil {
return Plan{}, err
}
status, err := inspectDestination(ctx, req.DestinationBackend, req.DestinationBundlePath)
if err != nil {
return Plan{}, err
@@ -91,6 +98,7 @@ func Build(ctx context.Context, req Request) (Plan, error) {
Action: action,
Reason: reason,
Force: action == ActionForceReplace,
PrimaryURL: primaryURL,
Outputs: outputs,
ExistingState: status.State,
}

View File

@@ -19,6 +19,7 @@ type DistributorState struct {
DestinationID string
PublishedAt time.Time
Source SourceState
Links *LinkState
Outputs []OutputFile
}
@@ -26,11 +27,16 @@ type SourceState struct {
Manifest bundle.Manifest
}
type LinkState struct {
PrimaryURL string
}
type OutputFile struct {
Path string
Kind string
SourcePath string
Transform string
URL string
SHA256 string
Size int64
}
@@ -42,6 +48,7 @@ type rawDistributorState struct {
DestinationID *string `json:"destination_id"`
PublishedAt *string `json:"published_at"`
Source *rawSourceState `json:"source"`
Links *rawLinkState `json:"links"`
Outputs []rawOutputFile `json:"outputs"`
}
@@ -49,11 +56,16 @@ type rawSourceState struct {
Manifest json.RawMessage `json:"manifest"`
}
type rawLinkState struct {
PrimaryURL string `json:"primary_url"`
}
type rawOutputFile struct {
Path *string `json:"path"`
Kind *string `json:"kind"`
SourcePath *string `json:"source_path"`
Transform string `json:"transform"`
URL string `json:"url"`
SHA256 *string `json:"sha256"`
Size *int64 `json:"size"`
}
@@ -112,6 +124,9 @@ func parseRaw(raw rawDistributorState) (DistributorState, error) {
return DistributorState{}, fmt.Errorf("state source.manifest: %w", err)
}
state.Source.Manifest = manifest
if raw.Links != nil {
state.Links = &LinkState{PrimaryURL: raw.Links.PrimaryURL}
}
if raw.Outputs == nil {
return DistributorState{}, fmt.Errorf("state outputs is required")
}
@@ -161,6 +176,7 @@ func parseOutput(index int, raw rawOutputFile) (OutputFile, error) {
Kind: *raw.Kind,
SourcePath: *raw.SourcePath,
Transform: raw.Transform,
URL: raw.URL,
SHA256: *raw.SHA256,
Size: *raw.Size,
}, nil
@@ -181,6 +197,7 @@ func (s DistributorState) MarshalJSON() ([]byte, error) {
DestinationID string `json:"destination_id"`
PublishedAt string `json:"published_at"`
Source sourceJSON `json:"source"`
Links *LinkState `json:"links,omitempty"`
Outputs []OutputFile `json:"outputs"`
}
return json.Marshal(stateJSON{
@@ -190,16 +207,25 @@ func (s DistributorState) MarshalJSON() ([]byte, error) {
DestinationID: s.DestinationID,
PublishedAt: s.PublishedAtString(),
Source: sourceJSON{Manifest: s.Source.Manifest},
Links: s.Links,
Outputs: s.Outputs,
})
}
func (l LinkState) MarshalJSON() ([]byte, error) {
type linkJSON struct {
PrimaryURL string `json:"primary_url,omitempty"`
}
return json.Marshal(linkJSON{PrimaryURL: l.PrimaryURL})
}
func (o OutputFile) MarshalJSON() ([]byte, error) {
type outputJSON struct {
Path string `json:"path"`
Kind string `json:"kind"`
SourcePath string `json:"source_path"`
Transform string `json:"transform,omitempty"`
URL string `json:"url,omitempty"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
@@ -208,6 +234,7 @@ func (o OutputFile) MarshalJSON() ([]byte, error) {
Kind: o.Kind,
SourcePath: o.SourcePath,
Transform: o.Transform,
URL: o.URL,
SHA256: o.SHA256,
Size: o.Size,
})

View File

@@ -29,6 +29,22 @@ func TestParseValidState(t *testing.T) {
}
}
func TestParseValidStateWithLinks(t *testing.T) {
body := strings.Replace(validStateJSON(t), `"outputs": [`, `"links": {"primary_url": "https://reports.example.com/archive/report.md"},`+"\n "+`"outputs": [`, 1)
body = strings.Replace(body, `"source_path": "report.md",`, `"source_path": "report.md",`+"\n "+`"url": "https://reports.example.com/archive/report.md",`, 1)
state, err := Parse([]byte(body))
if err != nil {
t.Fatalf("Parse() error = %v", err)
}
if state.Links == nil || state.Links.PrimaryURL != "https://reports.example.com/archive/report.md" {
t.Fatalf("links = %#v, want primary URL", state.Links)
}
if state.Outputs[0].URL != "https://reports.example.com/archive/report.md" {
t.Fatalf("output URL = %q", state.Outputs[0].URL)
}
}
func TestParseNormalizesPublishedAtOffset(t *testing.T) {
body := strings.Replace(validStateJSON(t), `"published_at": "2026-05-30T11:12:00Z"`, `"published_at": "2026-05-30T13:12:00+02:00"`, 1)
state, err := Parse([]byte(body))
@@ -145,6 +161,12 @@ func TestParseRejectsInvalidOutputMetadata(t *testing.T) {
"negative size": func(s *DistributorState) {
s.Outputs[0].Size = -1
},
"invalid output url": func(s *DistributorState) {
s.Outputs[0].URL = "file:///tmp/report.md"
},
"invalid primary url": func(s *DistributorState) {
s.Links = &LinkState{PrimaryURL: "file:///tmp/report.md"}
},
}
for name, mutate := range tests {
t.Run(name, func(t *testing.T) {
@@ -188,6 +210,36 @@ func TestMarshalNormalizesPublishedAtUTC(t *testing.T) {
}
}
func TestMarshalIncludesLinksWhenPresent(t *testing.T) {
source := validManifest(t)
state := DistributorState{
SchemaVersion: SchemaVersion,
PipelineID: "reports",
DestinationID: "archive",
PublishedAt: time.Date(2026, 5, 30, 11, 12, 0, 0, time.UTC),
Source: SourceState{Manifest: source},
Links: &LinkState{PrimaryURL: "https://reports.example.com/archive/report.md"},
Outputs: []OutputFile{{
Path: "report.md",
Kind: OutputKindSource,
SourcePath: "report.md",
URL: "https://reports.example.com/archive/report.md",
SHA256: source.Files[0].SHA256,
Size: source.Files[0].Size,
}},
}
data, err := json.Marshal(state)
if err != nil {
t.Fatalf("Marshal() error = %v", err)
}
if !strings.Contains(string(data), `"links":{"primary_url":"https://reports.example.com/archive/report.md"}`) {
t.Fatalf("json = %s, want links primary URL", data)
}
if !strings.Contains(string(data), `"url":"https://reports.example.com/archive/report.md"`) {
t.Fatalf("json = %s, want output URL", data)
}
}
func validStateJSON(t *testing.T) string {
t.Helper()
return validStateWithManifestJSON(t, manifestJSON(t))

View File

@@ -2,6 +2,7 @@ package state
import (
"fmt"
"net/url"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
@@ -28,6 +29,11 @@ func Validate(s DistributorState) error {
if err := validateEmbeddedManifest(s.Source.Manifest); err != nil {
return fmt.Errorf("state source.manifest: %w", err)
}
if s.Links != nil && s.Links.PrimaryURL != "" {
if err := validateStateURL(s.Links.PrimaryURL); err != nil {
return fmt.Errorf("state links.primary_url: %w", err)
}
}
if s.Outputs == nil {
return fmt.Errorf("state outputs is required")
}
@@ -63,6 +69,11 @@ func validateOutput(index int, output OutputFile) error {
if output.Kind == OutputKindGenerated && output.Transform == "" {
return fmt.Errorf("state outputs[%d].transform is required for generated output", index)
}
if output.URL != "" {
if err := validateStateURL(output.URL); err != nil {
return fmt.Errorf("state outputs[%d].url: %w", index, err)
}
}
if err := bundle.ValidateDigest(output.SHA256); err != nil {
return fmt.Errorf("state outputs[%d].sha256: %w", index, err)
}
@@ -71,3 +82,23 @@ func validateOutput(index int, output OutputFile) error {
}
return nil
}
func validateStateURL(value string) error {
parsed, err := url.Parse(value)
if err != nil {
return err
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("must use http or https")
}
if parsed.Host == "" {
return fmt.Errorf("must include a host")
}
if parsed.RawQuery != "" {
return fmt.Errorf("must not include a query string")
}
if parsed.Fragment != "" {
return fmt.Errorf("must not include a fragment")
}
return nil
}