11 Commits

76 changed files with 5863 additions and 2100 deletions

View File

@@ -2,7 +2,9 @@
`distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations.
It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars.
It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars or `index.html`.
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate complete local source bundles with the same manifest contract used by `distributor`.
Run the local example pipeline:
@@ -10,4 +12,4 @@ Run the local example pipeline:
go run ./cmd/distributor run --config examples/local-publish.yml
```
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Planning material lives under `docs/roadmap/`.
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Future and deferred work lives under `docs/roadmap/`.

View File

@@ -12,18 +12,22 @@ This discovers the example source bundle and publishes source files to `workspac
```sh
distributor [--help]
distributor version
distributor run [--config <path>] [--dry-run] [--force]
distributor validate <path>
distributor inspect <path>
distributor version [--format text|json]
distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
distributor validate [--format text|json] <path>
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor inspect [--format text|json] <path>
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor manifest create <bundle-path> --id <bundle-id> [options]
```
- `version`: prints the application name and version. Development builds print `distributor dev`.
- `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
- `validate`: validates a local source bundle directory or a local tree containing source bundles.
- `inspect`: validates local source bundles and prints normalized bundle metadata.
- `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source.
- `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source.
- `manifest create`: creates `manifest.json` for a local source bundle directory.
`validate` and `inspect` accept local paths only. `run` executes `local`, `ssh`, and `s3` backends.
`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured sources and destinations.
## Flag reference
@@ -35,13 +39,30 @@ All subcommands:
- `--help`, `-h`: print command-specific help.
Output-producing subcommands:
- `--format text|json`: output format. `text` is the default. Help and usage output are always text.
`run` flags:
- `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`.
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path; omitting the path returns a required-path error.
`validate` and `inspect` configured source flags:
- `--config <path>`: config file to load for source validation or inspection. Required in configured source mode.
- `--pipeline <id>`: pipeline source to validate or inspect. Required in configured source mode.
- `--bundle <path>`: source-root-relative bundle directory to validate or inspect instead of discovering every bundle under the source root.
`manifest create` flags:
- `--id <bundle-id>`: source bundle id. Required.
- `--file <path>`: bundle-relative file to include. Repeatable. If omitted, files are scanned recursively.
- `--created <time>`: RFC3339 source created timestamp. If omitted, the current UTC time is used.
- `--overwrite`: replace an existing `manifest.json`.
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path in local mode. Local paths cannot be combined with `--config`, `--pipeline`, or `--bundle`.
## Common workflows
@@ -57,6 +78,37 @@ Inspect a source bundle:
go run ./cmd/distributor inspect examples/source-bundle
```
Validate a configured source without opening destinations:
```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
```
Inspect one configured source bundle:
```sh
go run ./cmd/distributor inspect \
--config <config-path> \
--pipeline <pipeline-id> \
--bundle daily/2026-06-01
```
Create a manifest for a local producer bundle:
```sh
go run ./cmd/distributor manifest create <bundle-path> --id <bundle-id>
```
Create a manifest with explicit file order:
```sh
go run ./cmd/distributor manifest create <bundle-path> \
--id <bundle-id> \
--created 2026-06-01T11:00:00Z \
--file report.md \
--file summary.txt
```
Preview local publication without writing:
```sh
@@ -81,6 +133,12 @@ Preview local fan-out publication:
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
```
Preview local archive-plus-latest publication:
```sh
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
```
Preview a forced replacement before publishing:
```sh
@@ -89,7 +147,9 @@ go run ./cmd/distributor run --config <config-path> --dry-run --force
## Output
`run` prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the bundle path, destination id, destination backend, action, outputs, and reason. Actions include:
Text output is the default and is intended for humans.
`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the source bundle path, destination id, destination backend, action, outputs, and reason. Fixed path destinations also print `path_mapping=fixed target=.` to show that the selected bundle targets the destination backend root. Actions include:
- `publish_new`: destination has no managed state and is empty.
- `replace_older`: destination state is older than the source manifest.
@@ -100,10 +160,36 @@ go run ./cmd/distributor run --config <config-path> --dry-run --force
The command exits non-zero if any destination fails. Independent later destinations are still attempted.
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, and whether the run was a dry run.
Dry-run output for fixed path destinations prints a warning with the candidate count and selected source bundle. If a fixed path dry run plans a destructive replacement, it prints an additional warning that the destination root would be replaced.
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, whether the run was a dry run, and fixed path destinations.
JSON output writes exactly one JSON document to stdout:
```json
{
"schema_version": 1,
"command": "inspect",
"ok": true,
"warnings": [],
"result": {}
}
```
Warnings are objects in the top-level `warnings` array and are not printed again as text. Fatal setup errors, such as a missing config file or invalid arguments, write no JSON document and return a non-zero exit code with a text error on stderr.
`run --format json` returns partial results when destination failures occur after planning or execution begins. In that case stdout contains `ok: false`, a `result` with pipeline summaries, destination actions, final counters, and a top-level `errors` array; the process still exits non-zero.
Command-specific JSON results:
- `version`: application name and version.
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, path mapping markers, optional primary URLs, output records with optional URLs, final counters, warnings, and partial failure records.
## Diagnostics
Use `validate` before publication when a producer has written a new bundle. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle; use configured source mode when the bundle is already on an SSH or S3 source. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).

View File

@@ -54,7 +54,7 @@ pipelines:
## HTML Publication
To publish generated HTML from Markdown files:
To publish generated sidecar HTML from Markdown files:
```yaml
publish:
@@ -68,6 +68,77 @@ transform:
Sidecar generation writes `report.html` for `report.md`. It does not mutate the source bundle.
To publish a single Markdown file as `index.html`:
```yaml
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
When `mode: index` omits `input`, the source manifest must list exactly one Markdown file.
## Destination Path Mapping
Each destination chooses how source bundle paths map into that destination:
```yaml
path_mapping:
mode: preserve_relative
```
`preserve_relative` is the default. It publishes each discovered source bundle at the same path relative to the destination backend root. A source bundle at `daily/2026-06-01` publishes below `daily/2026-06-01` for that destination.
`fixed` publishes one selected source bundle directly at the destination backend root:
```yaml
destinations:
- id: latest-html
backend: local
path: /srv/www/reports/latest
path_mapping:
mode: fixed
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple candidates have the same timestamp, the source-root-relative bundle path in ascending order wins. Older candidates are not planned or written for that destination.
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
## Destination Links
Destinations can record public URLs for published outputs:
```yaml
links:
base_url: https://reports.example.com/archive
primary: auto
```
`links.base_url` is an absolute `http` or `https` URL corresponding to the destination backend root. It may include a path prefix, but it must not include a query string or fragment. Distributor does not infer public URLs from backend config.
`links.primary` selects the top-level primary URL stored in destination state:
- `auto`: prefer `index.html`, then generated HTML, then source outputs.
- `html`: use the first generated HTML output.
- `source`: use the first copied source output.
If a destination has no `links` block, no URL metadata is generated. If a primary policy has no matching output, per-output URLs are still recorded and the top-level primary URL is omitted.
Output URLs are built from `links.base_url`, the destination bundle path, and the output path using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
## Reference
Top level:
@@ -106,6 +177,9 @@ Destination:
- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
- `publish`: optional; defaults to source-only publication.
- `transform`: required only for generated HTML publication.
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
- `links.base_url`: optional links block; when present, `base_url` is required and must be an absolute HTTP or HTTPS URL without query string or fragment.
- `links.primary`: optional; defaults to `auto`. Accepted values are `auto`, `html`, and `source`.
- `transfer`: optional; defaults described below.
Accepted backend names:
@@ -164,7 +238,15 @@ Publish policy:
- `publish.source`: publish source artifacts.
- `publish.html`: publish generated HTML artifacts from Markdown source files.
At least one output type must be enabled. When `publish.html` is true, `transform.markdown_to_html.enabled` must be `true` and `transform.markdown_to_html.mode` must be `sidecar`.
At least one output type must be enabled. When `publish.html` is true, `transform.markdown_to_html.enabled` must be `true`.
Markdown-to-HTML transform:
- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for destinations with `publish.html: true`.
- `transform.markdown_to_html.mode`: optional; defaults to `sidecar`. Accepted values are `sidecar` and `index`.
- `transform.markdown_to_html.input`: optional source manifest path for `index` mode. It must identify a listed Markdown file.
`sidecar` mode renders each manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one selected Markdown file to `index.html` at the destination bundle path. Enabled Markdown-to-HTML config is rejected when `publish.html` is false, and `input` is valid only with `mode: index`.
Transfer policy:
@@ -184,8 +266,11 @@ Defaults are applied after YAML decoding and before validation:
- SSH `host_key_policy: accept-new`
- S3 `region: us-east-1`
- S3 `force_path_style: true`
- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
- `publish.source: true`
- `publish.html: false`
- `path_mapping.mode: preserve_relative`
- `links.primary: auto` when a `links` block is present and `primary` is omitted
- `transfer.on_destination_same: skip`
- `transfer.on_destination_older: replace`
- `transfer.on_destination_newer: skip`
@@ -216,6 +301,8 @@ Maintained examples live under [examples](../examples/):
- `local-to-local.yml`: minimal local config.
- `local-publish.yml`: runnable local source publication.
- `local-html.yml`: runnable local HTML publication.
- `local-index.yml`: runnable local `index.html` publication.
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
- `archive-and-latest.yml`: runnable local fan-out publication to an archive destination and a fixed latest destination.
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
- `s3-destination.yml`: environment-gated local-to-S3 publication example.

View File

@@ -12,7 +12,12 @@ Rendering uses `github.com/yuin/goldmark`. The exact dependency version is pinne
`internal/transform/markdown.New` constructs the renderer with `goldmark.New()` and no project-specific extensions or renderer options.
For each source bundle file ending in `.md`, the transform reads the Markdown source and generates an HTML sidecar in the same logical directory. The output path replaces the `.md` suffix with `.html`, so `report.md` produces `report.html`. Non-Markdown source files produce no Markdown outputs.
The transform supports two output modes:
- `sidecar`: reads each source bundle file ending in `.md` and generates an HTML sidecar in the same logical directory. The output path replaces the `.md` suffix with `.html`, so `report.md` produces `report.html`. Non-Markdown source files produce no Markdown outputs.
- `index`: renders one selected Markdown source to `index.html` at the destination bundle path.
In `index` mode, `transform.markdown_to_html.input` can name the source manifest path to render. If `input` is omitted, the manifest must list exactly one Markdown file. The selected input must be a safe relative source path, must be listed in the source manifest, and must end in `.md`.
Raw HTML embedded in Markdown is not passed through by the current renderer behavior. Tests allow Goldmark's disabled-or-escaped raw HTML output forms and reject literal script tags in generated HTML.
@@ -42,7 +47,7 @@ Generated outputs record:
Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select outputs, or choose transfer actions. Publish planning decides whether generated HTML is selected for a destination.
Only sidecar output mode is supported for current behavior.
Publish planning chooses the configured mode and input for each destination. Markdown rendering does not inspect destinations, publish files, write `.distributor.json`, or choose transfer actions.
## Tests
@@ -52,4 +57,4 @@ Before changing Markdown renderer behavior, inspect and run:
go test ./internal/transform/markdown
```
The tests cover sidecar naming, ignored non-Markdown files, raw HTML handling, deterministic output, digest metadata, and size metadata.
The tests cover sidecar naming, index input selection, ignored non-Markdown files, raw HTML handling, deterministic output, digest metadata, and size metadata.

View File

@@ -6,9 +6,9 @@
## Inputs and outputs
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, writes summary output when stdout is supplied, and returns an aggregated error if any destination fails.
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, output format, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, writes text or JSON output when stdout is supplied, and returns an aggregated error if any destination fails.
`Validate` and `Inspect` accept a local path. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
`Validate` and `Inspect` accept either a local path or one configured pipeline source. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
## Run flow
@@ -17,11 +17,12 @@ The runner:
1. loads config from the supplied path or `config.DefaultConfigPath`;
2. opens the configured source backend;
3. discovers validated bundles from the source root;
4. opens each destination backend independently;
5. builds a publish plan for each bundle and destination;
6. prints plan lines and records summary counters;
7. executes publish or replacement plans unless dry-run is enabled;
8. invokes the notifier after successful publish or replacement actions.
4. selects source bundles for each destination according to destination path mapping;
5. opens each destination backend independently;
6. builds publish plans for the selected bundle and destination combinations;
7. prints plan lines or JSON action records and records summary counters;
8. executes publish or replacement plans unless dry-run is enabled;
9. invokes the notifier after successful publish or replacement actions.
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out.
@@ -47,7 +48,7 @@ Stdout write errors are returned immediately because the caller's requested outp
`internal/app` coordinates packages but does not own manifest validation rules, destination state comparison, storage path rules, output planning, transform rendering, or backend-specific filesystem behavior.
`Validate` and `Inspect` are local path commands. Remote execution wiring currently belongs to `Run`.
Configured-source `Validate` and `Inspect` share source backend construction with `Run` and do not open destinations.
## Tests

View File

@@ -2,7 +2,7 @@
## Purpose
`internal/bundle` parses, discovers, and validates source bundles through the storage interface.
`internal/bundle` discovers and validates source bundles through the storage interface. The source manifest model, manifest parsing, manifest validation, path rules, digest calculation, and producer-side local writer come from `pkg/bundle` so producer-facing APIs and distributor validation share one manifest contract.
## Inputs and outputs
@@ -22,9 +22,9 @@ Each file requires `path`, `sha256`, and `size`. Digests must use lowercase `sha
## Validation
`ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest.
`pkg/bundle.ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest.
Storage-backed bundle validation additionally checks file existence, regular-file type, file size, and per-file SHA-256.
Storage-backed bundle validation in `internal/bundle` additionally checks file existence, regular-file type, file size, and per-file SHA-256 for configured storage backends.
The bundle digest is SHA-256 of a deterministic JSON array of file records in manifest order with fields `path`, `sha256`, and `size`.
@@ -38,11 +38,11 @@ Manifest parsing and validation fail before destination planning. Storage-backed
## Boundaries
Bundle code uses `internal/storage` and does not import concrete adapters. CLI local path support is wired in `internal/app`.
Internal bundle discovery uses `internal/storage` and does not import concrete adapters. Producer-side local filesystem manifest building, complete bundle writing, and validation belong to `pkg/bundle`. CLI local path support is wired in `internal/app`.
## Tests
Before changing bundle behavior, inspect tests under `internal/bundle`.
Before changing bundle behavior, inspect tests under `pkg/bundle` and `internal/bundle`.
## Invariants

View File

@@ -24,6 +24,9 @@ Defaults are applied in `ApplyDefaults`:
- SSH backend `port` defaults to `22`;
- SSH backend `host_key_policy` defaults to `accept-new`;
- destination publish policy defaults to source output only;
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
- destination path mapping defaults to `preserve_relative`;
- destination link primary policy defaults to `auto` when a `links` block is present;
- `transfer.on_destination_same` defaults to `skip`;
- `transfer.on_destination_older` defaults to `replace`;
- `transfer.on_destination_newer` defaults to `skip`;
@@ -31,11 +34,15 @@ Defaults are applied in `ApplyDefaults`:
## Validation responsibilities
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, and valid transfer actions.
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` mode. A publish policy must select source output, HTML output, or both.
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
Destination links are optional. When a `links` block is present, `base_url` is required, must use `http` or `https`, and must not include a query string or fragment. `primary` accepts `auto`, `html`, and `source`.
## Executable support boundary

View File

@@ -6,9 +6,9 @@
## Inputs and outputs
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, transformer resolver, transfer policy, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
Output is a plan with an action, reason, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
Output is a plan with an action, reason, optional primary URL, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
## Actions
@@ -22,9 +22,13 @@ Execution fails if a write, delete, state serialization, or context check fails.
## Boundaries
The current implementation publishes source files and Markdown-to-HTML sidecar outputs. Backend behavior is supplied through `internal/storage`; app runtime currently supplies local, SSH, and S3 backends.
The package publishes source files and Markdown-to-HTML outputs. Markdown sidecar mode writes same-directory `.html` outputs, and Markdown index mode writes `index.html`. Backend behavior is supplied through `internal/storage`; app runtime supplies local, SSH, and S3 backends.
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags or load config files.
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags, load config files, or choose which source bundles a destination receives.
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
When link config is present, publish planning builds per-output URLs from `links.base_url`, the destination bundle path, and each output path. `index.html` outputs use directory-style URLs. The primary URL is selected from planned outputs according to the destination primary policy.
## Safety
@@ -37,6 +41,8 @@ Before changing publish behavior, inspect tests under `internal/publish` and run
## Invariants
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
- URL generation uses URL path semantics and does not infer public URLs from backend configuration.
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
- Forced replacement is explicit per run and deletes only within the destination bundle path.
- Publish execution writes destination state after selected outputs are written.

View File

@@ -19,13 +19,15 @@ Input is JSON destination state plus the current source manifest, pipeline id, d
- `source.manifest`
- `outputs`
`distributor_version` is optional diagnostic metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
`distributor_version` is optional diagnostic metadata. `links` is optional URL metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
The embedded `source.manifest` is validated with the same source manifest rules used by `internal/bundle`.
## Outputs
Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`.
Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`. Outputs may record `url` when the destination has link generation configured.
The optional top-level `links.primary_url` records the selected primary URL for the published destination bundle. It is omitted when link generation is not configured or when the destination primary policy has no matching output.
## Comparison
@@ -33,7 +35,7 @@ Comparison outcomes cover absent destination state, unmanaged destination conten
## Failure behavior
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, invalid stored URLs, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
## Boundaries
@@ -48,5 +50,6 @@ Before changing destination state behavior, inspect tests under `internal/state`
- `.distributor.json` is the destination sentinel and state record.
- Embedded source manifests use the same validation rules as source bundles.
- Generated outputs always record a transform id.
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
- Comparison returns outcomes and reasons; it does not mutate storage.
- `distributor_version` is diagnostic metadata, not a comparison key.

View File

@@ -2,11 +2,11 @@
## Purpose
`internal/transform` defines generated publication artifacts. `internal/transform/markdown` implements Markdown-to-HTML sidecar generation.
`internal/transform` defines generated publication artifacts. `internal/transform/markdown` implements Markdown-to-HTML generation.
## Inputs and outputs
Inputs are a validated source bundle and source backend. Outputs include destination path, source path, transform id, generated bytes, SHA-256, and size.
Inputs are a validated source bundle, source backend, and transform options supplied by publish planning. Outputs include destination path, source path, transform id, generated bytes, SHA-256, and size.
## Registry
@@ -14,7 +14,7 @@ Inputs are a validated source bundle and source backend. Outputs include destina
## Markdown behavior
Markdown files ending in `.md` generate `.html` files in the same logical directory. Non-Markdown files do not generate outputs. Raw HTML embedded in Markdown is not passed through by the renderer.
Markdown sidecar mode renders files ending in `.md` to `.html` files in the same logical directory. Markdown index mode renders one selected manifest-listed Markdown file to `index.html`. Non-Markdown files do not generate sidecar outputs. Raw HTML embedded in Markdown is not passed through by the renderer.
Generated HTML is deterministic for the same source content and transform configuration.
@@ -22,7 +22,7 @@ See `docs/integrations/markdown.md` for the Goldmark integration contract.
## Failure behavior
Transform resolution fails when a requested transform id is not registered. Markdown rendering fails when the source file cannot be read or rendered. Publish planning fails when HTML output is requested and the selected transform produces no outputs for a bundle.
Transform resolution fails when a requested transform id is not registered. Markdown rendering fails when the source file cannot be read or rendered. Index input selection fails when the configured input is unsafe, not listed, not Markdown, or when no configured input can be inferred from exactly one manifest-listed Markdown file. Publish planning fails when HTML output is requested and the selected transform produces no outputs for a bundle.
## Boundaries
@@ -42,5 +42,6 @@ Before changing transform behavior, inspect tests under:
- Source bundle files are never mutated by transforms.
- Generated outputs record destination path, source path, transform id, SHA-256, and size.
- Markdown sidecar naming changes only the `.md` extension to `.html`.
- Markdown index mode always writes `index.html`.
- Non-Markdown source files do not generate Markdown outputs.
- Transform registration stays outside publish planning.

View File

@@ -26,12 +26,30 @@ Run the local HTML publication:
go run ./cmd/distributor run --config examples/local-html.yml
```
Run the local `index.html` publication:
```sh
go run ./cmd/distributor run --config examples/local-index.yml
```
Preview local fan-out publication:
```sh
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
```
Preview local archive-plus-latest publication:
```sh
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
```
Preview a run for automation:
```sh
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run --format json
```
Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control:
```sh
@@ -44,11 +62,21 @@ Preview an environment-gated S3 destination config after editing it for an S3-co
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
```
Validate one configured source without opening destinations:
```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
```
## Filesystem Layout
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
Destination bundle paths are configured per destination with `path_mapping.mode`.
The default mode, `preserve_relative`, preserves the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at that destination.
The `fixed` mode publishes one selected source bundle at the destination backend root. A fixed destination with local `path: /srv/www/reports/latest` writes outputs and `.distributor.json` directly under `/srv/www/reports/latest`. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp, with the source-root-relative bundle path as the deterministic tie-breaker.
The maintained local examples write under `workspace/`, which is ignored by Git.
@@ -64,19 +92,173 @@ Each published destination bundle contains `.distributor.json`. This file is the
- publication timestamp;
- source manifest used for publication;
- copied source output metadata;
- generated output metadata.
- generated output metadata;
- optional public URL metadata when destination links are configured.
`manifest.json` from the source bundle is not copied as destination state.
Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted.
## Go Producer Bundles
Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create complete local source bundles with the same path, digest, timestamp, and validation rules used by `distributor`.
Minimal producer-side bundle creation:
```go
manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
Root: outputDir,
ID: "reports.example.2026-05-30",
Files: []bundle.BundleFile{
{SourcePath: reportPath, Path: "report.md"},
{SourcePath: summaryPath, Path: "summary.txt"},
},
})
if err != nil {
return err
}
```
`WriteBundle` copies local producer files into a sibling temporary directory, writes `manifest.json`, validates the result, and promotes the completed bundle into place. It fails if `Root` already exists unless `Overwrite` is true. With overwrite enabled, it builds and validates the replacement before moving the existing root aside.
Use `BuildManifest` and `WriteManifest` when a producer already wrote all bundle files into the final root. `BuildManifest` can preserve an explicit file order, or `Scan: true` can recursively include regular files under `Root` in deterministic slash-path order. Scan mode includes dotfiles, excludes files named `manifest.json` or `.distributor.json`, and rejects symlinks.
Shell producers can create the same manifest through the CLI after writing bundle files:
```sh
go run ./cmd/distributor manifest create <bundle-path> --id reports.example.2026-05-30
go run ./cmd/distributor validate <bundle-path>
```
Use repeated `--file` flags to preserve a specific file order. If no `--file` flags are provided, the command scans the bundle directory recursively using the same filtering rules as `pkg/bundle.BuildManifest`.
## Static HTML Publication
Markdown-to-HTML publication can write sidecar files or a fixed `index.html`.
Use sidecar mode when each Markdown source should keep a matching HTML filename:
```yaml
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
Use index mode for static-site destinations that should serve a bundle through `index.html`:
```yaml
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
If `input` is omitted in index mode, the source manifest must list exactly one Markdown file. Generated HTML is recorded in `.distributor.json` with `kind: generated`, `source_path`, `transform: markdown_to_html`, digest, and size metadata.
## Archive And Latest Fan-Out
A pipeline can publish the same source to an archive destination and a stable latest destination:
```yaml
pipelines:
- id: reports
source:
backend: local
path: /var/spool/distributor/reports
destinations:
- id: archive
backend: local
path: /srv/reports/archive
path_mapping:
mode: preserve_relative
publish:
source: true
html: false
- id: latest-html
backend: local
path: /srv/www/reports/latest
path_mapping:
mode: fixed
links:
base_url: https://reports.example.com/latest
primary: auto
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
## Static Site URLs
Use destination `links` when a destination backend root corresponds to a public HTTP or HTTPS URL:
```yaml
links:
base_url: https://reports.example.com/archive
primary: auto
```
Distributor records URLs in `.distributor.json`; it does not publish notifications or infer URLs from local, SSH, or S3 backend fields.
For archive-style destinations, URLs include the destination bundle path. A source bundle under `daily/brentwood/2026-06-01` with `base_url: https://reports.example.com/archive` can produce:
```text
https://reports.example.com/archive/daily/brentwood/2026-06-01/report.html
```
For fixed destinations, URLs are rooted at `links.base_url`. A fixed HTML index destination with `base_url: https://reports.example.com/latest` records:
```text
https://reports.example.com/latest/
```
`index.html` outputs use directory-style URLs. Other outputs include their filename. The primary URL is selected from the published outputs using the destination `links.primary` policy.
## Source Validation and Inspection
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
Configured source validation is useful when producers write directly to SSH or S3 storage:
```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
go run ./cmd/distributor inspect --config <config-path> --pipeline <pipeline-id>
```
Use `--bundle <path>` to validate or inspect one source-root-relative bundle directory:
```sh
go run ./cmd/distributor validate \
--config <config-path> \
--pipeline <pipeline-id> \
--bundle daily/2026-06-01
```
For configured SSH sources, host key and authentication behavior matches `run`. For configured S3 sources, endpoint, bucket, prefix, region, path-style, explicit credential environment variables, and `secrets.directory` handling match `run`.
## Dry Runs
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.
Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`.
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication.
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, path mapping modes, optional link URLs, output records, final counters, and partial failure records. The summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
## Retry and Replacement Behavior
@@ -99,7 +281,7 @@ go run ./cmd/distributor run --config <config-path> --force
Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require `transfer.on_conflict: replace` plus `--force`. Newer destination state requires `transfer.on_destination_newer: replace` plus `--force`.
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For S3 destinations, deletion is constrained to the configured bucket and prefix plus the destination bundle prefix. Force is per run only and has no config default.
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For fixed destinations, the destination bundle path is the backend root, so forced replacement may clear that configured root but not its parent path, sibling directories, or anything outside the configured S3 bucket and prefix. Force is per run only and has no config default.
## Failure Handling
@@ -107,6 +289,8 @@ If one destination fails in a fan-out run, independent later destinations are st
Errors include the pipeline id, destination id, destination backend, and bundle path where applicable.
In JSON mode, destination failures after planning or execution begins are reported in the top-level `errors` array and in the run result while preserving a non-zero exit code. Fatal setup errors such as an unreadable config or invalid secrets directory write no JSON document.
If a write fails during publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.
After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.
@@ -127,7 +311,7 @@ S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optiona
When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. For fixed destinations, that prefix is the configured bucket plus optional `prefix`. Distributor does not manage bucket versioning or delete markers.
## Secrets Directory
@@ -138,9 +322,9 @@ secrets:
directory: /run/secrets/distributor
```
The directory is loaded during `run` before any source or destination backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the run fails before publication work starts.
The directory is loaded during `run` and configured-source `validate` or `inspect` before any backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` prints a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
## Caveats

View File

@@ -22,19 +22,21 @@ The current core workflow is:
2. open the source backend;
3. discover source bundles beneath the source root;
4. validate each source bundle and its `manifest.json`;
5. for each configured destination, inspect destination state;
6. compare source state to destination state;
7. build a publish plan;
8. optionally transform Markdown to HTML for that destination;
9. publish selected source and generated artifacts;
10. write `.distributor.json` as the destination sentinel/state file;
11. run the notification hook, which is a no-op in the MVP.
5. select the source bundle or bundles for each destination according to that destination's path mapping policy;
6. open each destination backend independently;
7. inspect destination state at the resolved destination bundle path;
8. compare source state to destination state;
9. build a publish plan that selects source files, generated files, destination paths, and optional public URLs;
10. optionally transform Markdown to HTML for that destination;
11. publish selected source and generated artifacts;
12. write `.distributor.json` as the destination sentinel/state file;
13. run the notification hook, whose default implementation is currently a no-op.
## Pipeline Model
A pipeline has exactly one source and one or more destinations.
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and notification behavior.
The source is discovered and validated once. Each destination has independent backend configuration, path mapping, publication policy, transform policy, public link policy, replacement behavior, state, and notification behavior.
The pipeline model is fan-out by design:
@@ -53,7 +55,7 @@ A source bundle is a directory containing `manifest.json`.
`manifest.json` is the sole producer-to-`distributor` contract. `distributor` must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest.
The MVP source manifest schema is intentionally minimal:
The source manifest schema is intentionally minimal:
```json
{
@@ -73,7 +75,7 @@ The MVP source manifest schema is intentionally minimal:
Required fields:
- `schema_version`: source manifest schema version. MVP value: `1`.
- `schema_version`: source manifest schema version. Current value: `1`.
- `id`: stable bundle identifier.
- `digest`: SHA-256 digest for the listed files.
- `created`: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
@@ -101,6 +103,7 @@ Each destination bundle path is managed by `.distributor.json`. This file is bot
- the normalized source manifest used for publication;
- metadata for copied source outputs;
- metadata for generated outputs, such as HTML files;
- optional URL metadata for published outputs;
- any additional metadata required by `distributor`.
A representative destination state file is:
@@ -127,14 +130,18 @@ A representative destination state file is:
]
}
},
"links": {
"primary_url": "https://reports.example.com/weather-daily/"
},
"outputs": [
{
"path": "report.html",
"path": "index.html",
"kind": "generated",
"source_path": "report.md",
"transform": "markdown_to_html",
"sha256": "sha256:...",
"size": 23456
"size": 23456,
"url": "https://reports.example.com/weather-daily/"
}
]
}
@@ -157,6 +164,8 @@ Transforms are configured per destination. A destination may receive source file
The MVP supports only Markdown-to-HTML transformation. HTML generation must not mutate the source bundle. Generated outputs must be deterministic from the source bundle and destination transform configuration, and must be recorded in `.distributor.json`.
Destination path mapping and public link generation are destination behavior. Source manifests do not declare where a bundle is published or which public URLs are recorded.
The application should distinguish:
- transform policy: how derived files are generated;
@@ -187,10 +196,11 @@ Avoid dependencies for small conveniences. Do not let external dependency types
Use this current layout unless the project has a documented reason to differ:
- `cmd/distributor`: application entrypoint only.
- `pkg/bundle`: public producer-facing source manifest model, digest logic, parsing, manifest building, complete local bundle writing, and local validation helpers.
- `internal/app`: application orchestration and top-level use cases.
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
- `internal/config`: configuration structs, defaults, loading, precedence, and validation.
- `internal/bundle`: source manifest parsing, source bundle discovery, source digest validation, and source bundle model.
- `internal/bundle`: storage-backed source bundle discovery and validation over the public manifest contract.
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
- `internal/adapters/local`: local filesystem backend.
@@ -228,17 +238,20 @@ Pipeline configuration should express:
- pipeline id;
- one source backend;
- one or more destinations;
- per-destination path mapping;
- per-destination publish policy;
- per-destination transform policy;
- per-destination public link policy;
- validation behavior;
- destination conflict/replacement behavior.
## Modules, Stages, and Registries
## Modules and Registries
Each major stage should have an explicit input/output contract:
Each major workflow step should have an explicit input/output contract:
- source discovery;
- source validation;
- destination bundle selection;
- destination state inspection;
- destination comparison;
- transform planning/execution;
@@ -297,8 +310,10 @@ Important tests include:
- relative path safety and path traversal rejection;
- destination `.distributor.json` parsing and comparison;
- same/older/newer/conflict publish decisions;
- destination bundle path mapping;
- destructive replacement safety checks;
- transform output planning and metadata recording;
- public URL planning and state metadata;
- dry-run output;
- local backend behavior with temporary directories;
- fake backend behavior for storage-facing core logic.

View File

@@ -6,10 +6,11 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
## Repository Layout
- `cmd/distributor`: executable entrypoint only.
- `pkg/bundle`: public producer-facing source manifest and local bundle writer helpers.
- `internal/app`: top-level use cases for `run`, `validate`, and `inspect`.
- `internal/cli`: standard-library command parsing, flags, help text, and command wiring.
- `internal/config`: YAML configuration structs, loading, defaults, and validation.
- `internal/bundle`: source bundle discovery, manifest parsing, digest calculation, and validation.
- `internal/bundle`: storage-backed source bundle discovery and validation using the public manifest contract.
- `internal/state`: destination `.distributor.json` parsing, validation, and comparison.
- `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
- `internal/adapters/local`: local filesystem backend.
@@ -18,15 +19,16 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
- `internal/storage/fake`: in-memory backend for tests.
- `internal/publish`: destination inspection, output planning, reconciliation, execution, managed cleanup, and explicit forced replacement.
- `internal/transform`: transform interface and registry.
- `internal/transform/markdown`: Markdown-to-HTML sidecar transform.
- `internal/transform/markdown`: Markdown-to-HTML transform.
- `internal/notify`: notification interface and current no-op notifier.
- `internal/testutil`: shared test fixtures. Production code must not import this package.
- `docs`: current user, operator, policy, internal, and roadmap documentation.
- `examples`: copyable example configs and source bundles.
Do not create new top-level package families such as `pkg`, `internal/stage`,
`internal/modules`, or service-specific adapter directories unless the
architecture policy or a current roadmap explicitly calls for them.
Do not create new top-level package families such as public `pkg/...` packages
beyond `pkg/bundle`, generic workflow containers, or service-specific adapter
directories unless the architecture policy or a current roadmap explicitly
calls for them.
## Common Commands
@@ -73,7 +75,7 @@ GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gom
- Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters.
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
- Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
- Use `bundle.ValidateManifest` for normalized source manifest semantics, including embedded source manifests in destination state.
- Use `pkg/bundle` for normalized source manifest semantics. Internal packages should reach those rules through `internal/bundle` when they also need storage-backed bundle discovery or validation.
- Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
- Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration.
- Do not import `internal/testutil` from production code.
@@ -127,8 +129,9 @@ When adding or changing commands or flags:
3. Add or update CLI tests in `internal/cli`.
4. Update `docs/cli.md` if syntax, flags, output expectations, or workflows change.
`validate` and `inspect` are local path commands. `run` loads configured
pipelines and executes local, SSH, and S3 backends.
`validate` and `inspect` support a local path shortcut and configured
source-only diagnostics. `run` loads configured pipelines and executes local,
SSH, and S3 backends.
## Storage Backends

View File

@@ -106,7 +106,7 @@ Recommended:
- `examples/`
- `docs/policy/development.md`
### Modular, staged, service-oriented, or orchestration application
### Modular, service-oriented, or orchestration application
Required:
- `docs/cli.md`, if CLI-based
@@ -175,7 +175,7 @@ It should include:
- dependency policy;
- how to add config fields;
- how to add CLI flags;
- how to add stages/modules/adapters, if applicable;
- how to add modules or adapters, if applicable;
- how to update examples;
- documentation update expectations.
@@ -216,7 +216,7 @@ Explain when commands are useful, not just their syntax.
**Audience:** administrators, operators
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
Required for applications that maintain state, support resume behavior, run multi-step workflows, write durable artifacts, use remote storage, or require recovery procedures.
It should cover:
@@ -248,7 +248,7 @@ Each entry should include:
**Audience:** developers, LLM coding agents
Required for modular, staged, service-oriented, or orchestration projects.
Required for modular, service-oriented, or orchestration projects.
This directory describes implemented internal components. It is not the roadmap.

View File

@@ -1,242 +0,0 @@
# Roadmap: CLI Output Policy
## Purpose
Define a shared CLI output policy before adding machine-readable JSON output to
`distributor` commands.
Current CLI output is human-readable text. That is appropriate as the default,
but upcoming features need a common machine-readable contract so each command
does not invent a separate JSON flag, envelope, warning policy, or error shape.
## Current Implementation Grounding
The current CLI has these output-producing commands:
- `version`;
- `run`;
- `validate`;
- `inspect`.
Current output is text-only. `run` has the most complex output because it can
print warnings, pipeline summaries, planned destination actions, partial
destination failures, and a final status line. `validate` and `inspect` are
currently local-path commands. `version` prints a single text line.
Planned `distributor manifest create`, remote `validate` and `inspect`, link
generation, and latest path destinations increase the need for structured
output that scripts can consume consistently.
## Goals
- Define one CLI-wide policy for text and JSON output.
- Preserve current human-readable text output as the default.
- Add JSON output consistently across output-producing commands when this
roadmap is implemented.
- Keep warning, error, and partial-failure behavior predictable.
- Avoid external dependencies; use Go's standard `encoding/json`.
## Non-Goals
- Do not add YAML, table, NDJSON, streaming JSON, or template output formats.
- Do not change source manifest, destination state, config, or backend schemas.
- Do not make help or usage output JSON.
- Do not turn ordinary fatal setup errors into structured JSON results.
- Do not add a root-global output flag in the first implementation.
## Shared CLI Policy
Each supported command should accept:
```sh
--format text
--format json
```
Policy:
- `text` is the default and preserves existing output unless a feature
explicitly changes text output.
- `json` writes exactly one JSON document to stdout.
- help and usage output remain text-only.
- invalid `--format` values are usage errors.
- `--format` is a per-command flag, not a root-global flag.
- a command must not accept `--format json` unless it emits the shared JSON
envelope for that command.
The first implementation should add JSON support for all current
output-producing commands rather than leaving a mixed CLI where some commands
support `--format` and others do not.
## Stdout, Stderr, Warnings, and Errors
Text mode keeps the current behavior:
- normal command output goes to stdout;
- warnings may be printed as text;
- command errors are printed to stderr by CLI error handling.
JSON mode:
- successful commands write one JSON document to stdout;
- warnings are included in a top-level `warnings` array and are not duplicated
to stderr;
- fatal setup errors that prevent construction of a result write text errors to
stderr, write no JSON stdout, and exit non-zero;
- commands with meaningful partial results may write a JSON document with
`ok: false`, structured failure details, and a non-zero exit status.
`run --format json` should use the partial-result behavior when planning or
execution has begun and one or more destinations fail. This lets automation
inspect successful actions, failed actions, warnings, and final counters even
when the process exits non-zero.
## JSON Envelope
All JSON-mode commands should use a common top-level envelope:
```json
{
"schema_version": 1,
"command": "inspect",
"ok": true,
"warnings": [],
"result": {}
}
```
Envelope fields:
- `schema_version`: integer version of the CLI JSON output schema.
- `command`: command name, using the public CLI command path where useful, such
as `manifest create`.
- `ok`: boolean success indicator for the command result.
- `warnings`: array of structured warning objects.
- `result`: command-specific result object.
- `errors`: optional array of structured error objects for commands that can
return partial results.
Field rules:
- use stable snake_case field names;
- use RFC3339 timestamps;
- use numeric JSON values for sizes and counts;
- use slash-separated logical paths for bundle, source, destination, and output
paths;
- never include secret values;
- keep command-specific data under `result`;
- add fields compatibly where practical.
Warning objects should include at least:
```json
{
"message": "secret OBJECT_STORAGE_KEY ignored because the real environment already has that variable"
}
```
Error objects for partial results should include enough context for automation
and troubleshooting, such as pipeline id, destination id, backend, bundle path,
and message when those values are available.
## Command Adoption
`version`:
- text mode keeps the current version string;
- JSON mode reports application name and version under `result`.
`validate`:
- text mode keeps the current validation summary;
- JSON mode reports bundle count and selected bundle identifiers;
- future remote validation uses the same envelope and adds pipeline/source
context under `result`.
`inspect`:
- text mode keeps human-readable bundle metadata;
- JSON mode reports normalized bundle metadata, including source-relative path,
bundle id, created timestamp, digest, file count, total size, and file
records.
`run`:
- text mode keeps the current progress and final status style unless a future
feature intentionally changes it;
- JSON mode reports warnings, pipeline summaries, destination action records,
output records, final counters, dry-run status, and partial failure records;
- JSON mode may write `ok: false` with partial results and still exit non-zero.
Future `manifest create`:
- should use `--format text|json`;
- text mode should keep concise human success output;
- JSON mode should report the generated manifest summary under `result`;
- the command should not introduce a separate `--json` flag.
## Relationship To Other Roadmaps
`distributor manifest create` should depend on this roadmap for JSON summary
output and should not add a command-specific JSON flag.
Remote `validate` and `inspect` should use this policy for machine-readable
source validation and inspection.
Link generation should expose primary links through JSON command output only
after this policy is implemented or selected for implementation.
Latest path destinations should report fixed-path selection and destructive
replacement summaries through the same `run --format json` result model.
## Testing Expectations
Suggested coverage:
- CLI parsing accepts `--format text` and `--format json` for supported
commands;
- CLI parsing rejects invalid `--format` values as usage errors;
- help and usage output remain text-only;
- each JSON-capable command emits exactly one valid JSON document to stdout on
success;
- text mode preserves existing output;
- JSON-mode warnings appear in `warnings` and are not duplicated to stderr;
- fatal setup errors write no JSON stdout and return non-zero;
- `run --format json` emits partial-result JSON with `ok: false` and exits
non-zero when one or more destination failures occur after planning begins;
- JSON output uses RFC3339 timestamps, numeric sizes and counts, and
slash-separated logical paths;
- no JSON output includes secret values;
- documentation consistency checks find no unsupported `--json` references.
## Documentation Updates After Implementation
- Update `docs/cli.md` with the shared `--format text|json` policy.
- Update command examples only for implemented JSON behavior.
- Update `docs/operations.md` where JSON output materially improves automation
workflows.
- Update `docs/troubleshooting.md` only for implemented JSON-mode recovery
behavior.
- Update relevant roadmap files when JSON output is no longer deferred.
Keep this roadmap under `docs/roadmap/` until implemented.
## Decisions
- Use per-command `--format text|json`; do not introduce `--json`.
- Keep `text` as the default for backward compatibility.
- Implement JSON support for all current output-producing commands when this
roadmap is selected.
- Keep help and usage output text-only.
- Put JSON-mode warnings in the top-level `warnings` array.
- Allow `run --format json` to emit partial-result JSON with `ok: false` and a
non-zero exit status.
- Keep fatal setup errors as text stderr with no JSON stdout.
## Future Work
- Consider a root-global output flag only if the command parser is later
refactored around shared root options.
- Consider additional formats only if a concrete consumer requires them.
- Consider a versioned JSON schema reference after the first JSON-capable
release.

View File

@@ -1,160 +0,0 @@
# Roadmap: HTML Index Mode
## Purpose
Add a Markdown-to-HTML `index` mode that renders one selected Markdown artifact
to `index.html` at the destination bundle path.
This feature improves static-site UX and gives link generation and latest path
destinations a clean directory-style output to prefer.
## Current Implementation Grounding
Current Markdown transformation lives in `internal/transform/markdown`. It
renders every manifest-listed `.md` file to a sidecar `.html` file:
```text
report.md -> report.html
```
Current config validation accepts only `markdown_to_html.mode: sidecar`.
`publish.PlanOutputs` asks the configured transformer for generated outputs,
then destination state records generated outputs with path, kind, source path,
transform name, SHA-256, and size. Current destination state has no field for a
transform mode beyond the existing transform string.
## Goals
- Preserve existing sidecar behavior as the default and supported explicit mode.
- Add `markdown_to_html.mode: index`.
- In index mode, render exactly one Markdown input to `index.html`.
- Record `index.html` as a normal generated output in `.distributor.json`.
- Keep source manifests unchanged.
- Keep link generation and latest paths straightforward without making them
dependencies.
## Non-Goals
- Do not add collection index pages.
- Do not add multi-page static-site generation.
- Do not add feeds, notifications, or link generation in this feature.
- Do not require producers to name a file `index.md`.
- Do not add custom output filenames in v1.
## Configuration Shape
Extend existing destination-level transform config:
```yaml
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
Mode semantics:
- `sidecar`: existing behavior; render each manifest-listed Markdown file to a
same-directory `.html` file.
- `index`: render one selected Markdown file to `index.html` at the destination
bundle path.
`sidecar` remains the current/default mode. `index` is the proposed addition.
## Input Selection
Index mode should choose the Markdown input deterministically:
1. If `transform.markdown_to_html.input` is configured, use that manifest-listed
Markdown file.
2. If no input is configured and the source manifest lists exactly one Markdown
file, use that file.
3. If no input is configured and there are zero or multiple Markdown files, fail
planning with a clear error.
The configured input path must be a safe relative source path, must be listed in
the source manifest, and must end in `.md`.
Input selection happens during planning because it depends on the validated
source manifest, not only static config.
## Output Semantics
Index mode always writes:
```text
index.html
```
relative to the destination bundle path.
The generated output should use current state model terms:
- `path`: `index.html`;
- `kind`: `generated`;
- `source_path`: selected Markdown file;
- `transform`: `markdown_to_html`;
- `sha256` and `size`: digest and byte size of generated HTML.
Replacement, skip, cleanup, and force behavior should treat `index.html` like
any other distributor-managed generated output.
Config validation should reject enabled Markdown-to-HTML transform
configuration when `publish.html` is false. That keeps publish policy and
transform intent aligned and avoids silently accepting unused transform config.
## Relationship To Other Roadmaps
Link generation should recognize `index.html` and produce directory-style URLs,
but link generation must not be required for index mode.
Latest path destinations work without index mode, but fixed latest destinations
produce nicer stable URLs when they publish `index.html`.
`manifest create`, `pkg/bundle`, and remote validation remain source-bundle
features and should not depend on transform mode.
## Testing Expectations
Suggested coverage:
- config validation accepts `mode: sidecar` and `mode: index`;
- current sidecar behavior remains unchanged;
- index mode accepts explicit input;
- index mode chooses the only Markdown file when no input is configured;
- planning fails when index mode has zero or multiple Markdown candidates
without explicit input;
- planning fails when explicit input is unsafe, not listed, or not Markdown;
- index mode emits `index.html`;
- state records `index.html` as a generated output;
- source-only publication does not write `index.html`;
- dry-run reports `index.html` without writing it;
- replacement safely updates prior generated `index.html`.
## Documentation Updates After Implementation
- Update `docs/config.md` with Markdown-to-HTML modes and `input`.
- Update `docs/integrations/markdown.md` with sidecar and index behavior.
- Update `docs/operations.md` with a static-site example.
- Add or update examples only for implemented behavior.
Keep this roadmap under `docs/roadmap/` until implemented.
## Decisions
- Enabled Markdown-to-HTML transform config is rejected when `publish.html` is
false.
- Generated output metadata keeps `transform: markdown_to_html`; the output
path and destination config distinguish sidecar from index behavior.
- Future multi-page or collection index generation should be a separate
transform, not an expansion of this single-input index mode.
## Future Work
- Add a separate collection or site-index transform if distributor later needs
multi-page aggregation.
- Consider richer transform metadata only if future state consumers need more
than the transform name and output path.
- Consider custom index output names only if fixed `index.html` proves too
limiting for real deployments.

View File

@@ -1,10 +1,8 @@
# Implementation Roadmap
# Roadmap
This roadmap records current implementation status and deferred work for
`distributor`. Implemented behavior is documented in the user, operator, and
internal documentation listed below.
Canonical current-behavior docs:
This directory contains only future, deferred, or aspirational work for
`distributor`. Implemented behavior is documented in the current user,
operator, internal, policy, integration, and example documentation:
- `README.md`
- `docs/cli.md`
@@ -16,338 +14,89 @@ Canonical current-behavior docs:
- `docs/policy/`
- `examples/`
Future, planned, or aspirational behavior belongs under `docs/roadmap/` until
it is implemented.
## Current State
`distributor` is ready for routine use against producer pipelines using the
implemented local, SSH/SFTP, and S3-compatible backends.
## Active Roadmap
The active roadmap implements the focused feature roadmaps in an order intended
to minimize later rewrites. The sequence establishes shared CLI output before
new CLI commands, establishes producer bundle APIs before manifest creation,
and implements destination path mapping before link generation so URL
construction does not need to be reworked later.
Each stage is sized for one implementation prompt. Future behavior stays under
`docs/roadmap/` until the stage is implemented and current-behavior docs are
updated.
### Stage 1: CLI Output Policy
Goal: implement the shared CLI output contract in
`docs/roadmap/cli_output_policy.md`.
Implementation scope:
- add per-command `--format text|json` to current output-producing commands:
`version`, `validate`, `inspect`, and `run`;
- keep `text` as the default and preserve existing text output unless the
policy explicitly requires a change;
- implement the shared JSON envelope, warnings array, fatal setup error
behavior, and `run` partial-result JSON with `ok: false`;
- keep help and usage output text-only.
Documentation updates after implementation:
- update `docs/cli.md` with the shared `--format text|json` policy;
- update `docs/operations.md` where JSON output materially improves automation
workflows;
- update `docs/troubleshooting.md` only for implemented JSON-mode recovery
behavior.
Tests:
- CLI parsing accepts `--format text` and `--format json`;
- invalid `--format` values are usage errors;
- each JSON-capable command emits exactly one valid JSON document on success;
- JSON-mode warnings appear in `warnings` and are not duplicated to stderr;
- fatal setup errors write no JSON stdout and return non-zero;
- `run --format json` emits partial-result JSON and exits non-zero when
destination failures occur after planning begins.
Completion criteria: all current output-producing commands support the shared
policy, text output remains backward compatible, and `go test ./...` passes.
### Stage 2: Public Bundle Package Core
Goal: implement the core and manifest-building parts of
`docs/roadmap/public_bundle_package.md`.
Implementation scope:
- introduce `pkg/bundle` with the public source manifest model, schema version,
digest logic, parsing, validation, explicit file-list building, scan-based
building, and zero-`Created` defaulting to current UTC;
- implement the locked Stage 2 exported API symbols defined in
`docs/roadmap/public_bundle_package.md`;
- update internal validation to consume the shared implementation without
behavior drift;
- keep destination state, publish planning, storage backends, transforms,
notifications, and config internal.
Documentation updates after implementation:
- add package documentation for `pkg/bundle`;
- update `README.md` and `docs/operations.md` only for implemented Go producer
support;
- update `docs/internal/bundle.md` if internal ownership changes.
Tests:
- public package tests cover explicit lists, scan mode, ordering, digest
calculation, parsing, validation, unsafe paths, symlinks, and zero `Created`;
- internal bundle tests continue to pass against the shared implementation;
- public examples compile where practical.
Completion criteria: public and internal validation use one manifest contract,
and tests prove identical semantics.
### Stage 3: Public Bundle Local Writer
Goal: implement the local bundle writer portion of
`docs/roadmap/public_bundle_package.md`.
Implementation scope:
- add producer-side local bundle writing with staging, atomic filesystem
operations where practical, and best-effort restore on overwrite failure;
- implement the locked Stage 3 exported API symbols defined in
`docs/roadmap/public_bundle_package.md`;
- keep the writer filesystem-local and producer-focused;
- do not expose distributor storage backends or publication behavior through
the public package.
Documentation updates after implementation:
- document the writer API in `pkg/bundle`;
- update producer workflow docs only for implemented behavior.
Tests:
- writer creates a complete valid local bundle through staged promotion;
- writer output validates through distributor's normal validation path;
- overwrite and failure behavior avoid leaving a completed bundle path without
a valid manifest where practical.
Completion criteria: producers can create complete valid local bundles through
the public API without hand-writing manifest files.
### Stage 4: Manifest Create CLI
Goal: implement `docs/roadmap/manifest_create.md`.
Implementation scope:
- add `distributor manifest create` over `pkg/bundle`;
- support explicit-first `--file` selection, recursive scan fallback,
`--overwrite`, optional `--created`, and shared `--format text|json`;
- use temp-and-rename replacement for `manifest.json` where practical;
- do not duplicate bundle contract logic in CLI or app code.
Documentation updates after implementation:
- update `docs/cli.md` with command syntax, flags, output, and examples;
- update `docs/operations.md` with a producer workflow;
- cross-reference `pkg/bundle` for Go producers.
Tests:
- command creates valid manifests for explicit and scanned files;
- ordering, dotfile inclusion, metadata exclusion, symlink rejection, and
overwrite behavior match the roadmap;
- text and JSON output follow the shared CLI output policy.
Completion criteria: the command writes valid `manifest.json`, supports
text/JSON output, and generated manifests validate through distributor.
### Stage 5: Remote Validate and Inspect
Goal: implement `docs/roadmap/remote_validate_inspect.md`.
Implementation scope:
- extend `validate` and `inspect` with mutually exclusive local-path and
`--config --pipeline` modes;
- reuse `run` source backend construction and secrets resolution;
- support configured local, SSH, and S3 sources;
- preserve `--pipeline` as required in config mode;
- keep the commands source-only and do not open destinations.
Documentation updates after implementation:
- update `docs/cli.md` with local and config-driven syntax;
- update `docs/operations.md` with remote validation examples;
- update `docs/troubleshooting.md` for common configured-source failures.
Tests:
- existing local behavior remains unchanged;
- config-mode validation and inspection work for local and fake/app-level
source backends;
- SSH and S3 behavior is covered by focused unit tests and existing opt-in
integration patterns;
- JSON output follows the shared CLI output policy.
Completion criteria: local behavior is stable, configured local/SSH/S3 sources
can be validated and inspected, and no destination backend is opened.
### Stage 6: HTML Index Mode
Goal: implement `docs/roadmap/html_index_mode.md`.
Implementation scope:
- add `markdown_to_html.mode: index` with fixed `index.html` output;
- keep `sidecar` as the default and existing explicit mode;
- implement deterministic input selection through explicit input or exactly one
manifest-listed Markdown file;
- reject enabled Markdown-to-HTML transform config when `publish.html` is
false;
- record generated state metadata with `transform: markdown_to_html`.
Documentation updates after implementation:
- update `docs/config.md` with Markdown-to-HTML modes and input selection;
- update `docs/integrations/markdown.md`;
- update `docs/operations.md` with an implemented static-site example where
useful.
Tests:
- sidecar behavior remains unchanged;
- index mode handles explicit input, single Markdown fallback, ambiguous input,
unsafe input, state metadata, dry-run, source-only publication, and output
collisions.
Completion criteria: source publication, sidecar mode, index mode, state
metadata, dry-run, and collision behavior are all covered.
### Stage 7: Latest Path Destinations
Goal: implement `docs/roadmap/latest_paths.md`.
Implementation scope:
- add destination-level `path_mapping.mode` with default `preserve_relative`
and new `fixed`;
- make destination bundle path mapping destination-local;
- for fixed destinations, select only the newest discovered bundle per
destination before planning writes;
- add fixed-path dry-run warnings and summary counts;
- allow `--force` for fixed backend roots while keeping deletion bounded to the
configured backend root.
Documentation updates after implementation:
- update `docs/config.md` with `path_mapping`;
- update `docs/operations.md` with archive-plus-latest fan-out examples;
- update `docs/cli.md` if dry-run output gains fixed-path indicators.
Tests:
- omitted and explicit `preserve_relative` match existing behavior;
- fixed destinations publish at local, SSH, and S3 backend roots;
- newest-only selection is deterministic;
- older discovered bundles are not planned or written to fixed destinations;
- dry-run reports candidate count, selected bundle, and destructive replacement
warnings;
- force remains bounded at the backend root.
Completion criteria: fixed destination roots work across implemented backends,
archive-style destinations remain unchanged, and destructive behavior is clear
in dry-run output.
### Stage 8: Link Generation Support
Goal: implement `docs/roadmap/link_generation.md`.
Implementation scope:
- add destination-level `links.base_url` and `links.primary`;
- generate per-output and primary URLs using URL semantics, not filesystem or
storage joins;
- handle `index.html` as a directory-style URL;
- store optional URL metadata under destination state schema version 1 while
pre-release;
- expose links through JSON command output only where supported by the CLI
output policy.
Documentation updates after implementation:
- update `docs/config.md` with destination `links` fields;
- update `docs/operations.md` with static-site URL examples;
- update `docs/internal/state.md` for URL metadata;
- update `docs/cli.md` only for implemented CLI link output.
Tests:
- config validation accepts valid links and rejects invalid schemes, query
strings, and fragments;
- nested bundle paths and fixed latest paths generate correct URLs;
- `index.html` omits the filename;
- primary selection is deterministic;
- destinations without `links` produce no URL metadata.
Completion criteria: URL generation respects archive and fixed path mapping,
state records optional URL metadata when configured, and unconfigured
destinations remain unchanged.
### Stage 9: Roadmap Closeout
Goal: remove completed roadmap drift after Stages 1-8 are implemented.
Implementation scope:
- remove or rewrite completed roadmap files whose behavior is fully documented
in current docs;
- ensure current docs describe implemented behavior;
- keep `docs/roadmap/` focused only on remaining future work.
Documentation updates after implementation:
- update `README.md`, `docs/cli.md`, `docs/config.md`,
`docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`,
`docs/integrations/markdown.md`, and examples only where implemented
behavior requires it.
Tests:
- run documentation consistency searches for completed-feature language that
still appears only as future work;
- run focused tests for any examples or docs backed by tests;
- run `go test ./...` if behavior docs and examples changed with code.
Completion criteria: completed features are documented as current behavior, and
`docs/roadmap/` contains only future or deferred work.
## Deferred Work
These items are not implemented and should stay out of current-behavior docs
until a roadmap entry is selected and implemented:
- external notification adapters;
- warning-only digest mismatch handling;
- additional auth mechanisms beyond the implemented SSH and S3 credential
paths;
- compatibility parsing for legacy SSH URI config;
- broad recursive destination deletion outside managed bundle paths;
- concurrent fan-out publishing;
- streaming, resumable, or multipart S3 uploads;
- cloud-provider-specific IAM integration docs;
- repository-managed packaging, release, and deployment automation.
`distributor` currently supports local, SSH/SFTP, and S3-compatible source and
destination backends; producer bundle creation through `pkg/bundle` and
`distributor manifest create`; configured source validation and inspection;
Markdown sidecar and `index.html` publication; archive and fixed destination
path mapping; destination link metadata; shared text/JSON CLI output; and
managed destination replacement behavior.
## Future Work
These items are not implemented. They should not be documented as current
behavior outside `docs/roadmap/` unless a future implementation adds them.
### CLI And Status Output
- Add a root-global output flag only if the command parser is later refactored
around shared root options.
- Add output formats beyond `text` and `json` only if a concrete consumer
requires them.
- Add a versioned JSON schema reference after the first JSON-capable release.
- Add destination-state inspection behind an explicit flag such as
`--with-destinations` if operators need fan-out status diagnostics from
`inspect`.
- Add additional status or inspection presentation for destination primary
links beyond the current `run --format json` result model.
### Producer Workflows
- Add a no-write manifest creation mode, such as writing manifest JSON to
stdout, if producer pipelines need to capture manifests directly.
- Add broader producer workflow helpers, such as richer ignore rules or
template scaffolding, if real producer use cases require them.
- Add remote or storage-backed producer writers only if producer applications
need to assemble bundles outside the local filesystem.
### Publication And Transform Behavior
- Add a separate collection or site-index transform if distributor needs
multi-page aggregation.
- Add richer transform metadata only if future state consumers need more than
the transform name and output path.
- Add custom HTML index output names only if fixed `index.html` is too limiting
for real deployments.
- Add richer fixed-destination source selection policies if deployments need
something other than newest-by-`created`.
- Add stricter handling for equal latest timestamps if timestamp ties become
common in producer workflows.
- Add higher-level status or approval workflows for fixed-root replacements if
dry-run output is not enough operational protection.
- Add richer link policies only if `auto`, `html`, and `source` prove
insufficient.
### State And Compatibility
- Define a post-release destination state schema bump policy before introducing
materially incompatible state changes.
- Add warning-only digest mismatch handling only if an operator workflow needs
publication to continue after validation failures.
- Add compatibility parsing for legacy SSH URI config only if migration support
is required.
### Backends, Security, And Deployment
- Add authentication mechanisms beyond the implemented SSH agent/key and S3
credential paths only when a concrete backend workflow requires them.
- Add broad recursive destination deletion outside managed bundle paths only if
a future design can preserve the current safety boundary.
- Add concurrent fan-out publishing only if runtime profiling shows it is
needed.
- Add streaming, resumable, or multipart S3 uploads only if object sizes make
the current write path insufficient.
- Add cloud-provider-specific IAM integration docs only when the repository
includes tested provider-specific behavior.
- Add repository-managed packaging, release, and deployment automation when the
release process is ready to be standardized.
## Roadmap Maintenance
When adding future roadmap work:
- describe user-visible behavior and safety boundaries;
- define which existing docs must change after implementation;
- define which current docs must change after implementation;
- keep examples secret-free and runnable or clearly environment-gated;
- avoid workflow labels in production code, tests, config fields, and user
documentation;
- keep workflow labels out of production code, tests, config fields, and
user-facing documentation;
- run focused tests for the changed behavior and `go test ./...` for
cross-package changes.

View File

@@ -1,212 +0,0 @@
# Roadmap: Latest Path Destinations
## Purpose
Support stable "latest" publication paths as normal fan-out destinations.
A latest destination republishes the current winning source bundle to a fixed
destination path such as `/weather/latest/`, while archive destinations preserve
source-relative bundle paths such as `/weather/daily/brentwood/2026-06-01/`.
This feature should be implemented before link generation so URL construction
can use final destination path semantics. It is operationally destructive and
benefits from index-mode HTML, although it must not require index mode or link
generation.
## Current Implementation Grounding
Current run behavior computes the destination bundle path as the source bundle
path relative to the source root. Publication then writes selected outputs and
`.distributor.json` below that destination bundle path.
Backend roots differ by backend:
- local and SSH/SFTP use configured `path` as the backend root;
- S3 uses configured bucket plus optional `prefix` as the backend root.
Destination comparison uses `.distributor.json` at the destination bundle path.
Replacement and skip decisions are based on the normalized source manifest
recorded in destination state. Unmanaged non-empty destinations fail unless
explicit force behavior is requested.
## Goals
- Add destination-local path mapping.
- Preserve source-relative destination paths as the default.
- Add fixed path mapping for latest-style destinations.
- For fixed destinations, select only the newest discovered source bundle for
publication.
- Make fixed mapping work for local, SSH/SFTP, and S3 backends.
- Continue using normal destination state comparison and replacement rules.
- Keep link generation and index-mode HTML complementary, not required.
## Non-Goals
- Do not add symlink-based latest behavior.
- Do not add feed or index generation.
- Do not change source manifest schema.
- Do not make producers responsible for latest publication.
- Do not add domain-specific latest selection rules.
## Configuration Shape
Add destination-level path mapping:
```yaml
path_mapping:
mode: preserve_relative
```
Modes:
- `preserve_relative`: existing/default behavior.
- `fixed`: publish selected outputs directly at the destination backend root.
Example fixed destination:
```yaml
destinations:
- id: latest-html
backend: ssh
host: web.example.com
user: deploy
path: /srv/www/weather/latest
path_mapping:
mode: fixed
publish:
source: false
html: true
```
For S3, the fixed destination root is the configured bucket plus `prefix`.
## Path Mapping Semantics
`preserve_relative`:
```text
destination bundle path = source-root-relative bundle path
```
`fixed`:
```text
destination bundle path = ""
```
That empty logical destination bundle path means the destination backend root.
Storage writes still use normal backend-rooted logical paths for outputs and
state.
## Newest Bundle Selection
`preserve_relative` destinations should continue publishing every selected
source bundle independently.
`fixed` destinations should publish only one bundle per run: the newest
discovered source bundle selected for that destination. Newest selection should
be deterministic:
1. choose the bundle with the greatest source manifest `created` timestamp;
2. if multiple bundles have the same greatest `created` timestamp, use the
source-root-relative bundle path ascending as a tie-breaker.
The selected bundle then uses the existing destination comparison, skip,
replacement, cleanup, and force rules. Older discovered bundles should not be
planned or written to that fixed destination in the same run.
Dry-run output should identify the number of fixed-destination candidates and
the selected source bundle so operators can see which bundle would become
latest.
## Safety Rules
Fixed path mapping is more destructive than archive-style publication because
newer bundles replace prior contents at the same destination path.
Required behavior:
- dry-run output must identify fixed path mapping and planned replacement;
- dry-run output must include an extra fixed-path warning or summary count when
destructive replacement is possible;
- replacement should delete only managed outputs recorded in valid destination
state when possible;
- unmanaged non-empty fixed destinations must fail unless force is explicitly
requested;
- `--force` may be used when fixed mapping targets the backend root, but force
replacement must remain bounded to the configured destination backend root and
must never delete above it;
- fixed mapping to the backend root requires especially clear dry-run reporting.
## Relationship To Other Roadmaps
HTML index mode is useful for fixed web destinations because `index.html`
supports stable directory-style URLs, but sidecar HTML and source-only outputs
should still work.
Link generation should use fixed path semantics so latest URLs are based on the
fixed destination root, not the original source-relative archive path.
Producer manifest creation and remote validation are independent of destination
path mapping.
## Implementation Stages
1. Add destination-level `path_mapping` config validation with
`preserve_relative` as the default and `fixed` as the new mode.
2. Refactor run planning so destination bundle path mapping is destination-local
and can be evaluated before writes.
3. Add fixed-destination newest selection so each fixed destination receives
only the newest discovered source bundle.
4. Integrate fixed mapping with destination comparison, managed deletion, force
replacement, and dry-run reporting.
5. Update local, SSH, and S3 tests to verify fixed root behavior under each
backend root model.
6. Update current-behavior documentation after implementation.
## Testing Expectations
Suggested coverage:
- omitted `path_mapping` preserves existing behavior;
- `preserve_relative` explicitly matches existing behavior;
- `fixed` publishes outputs and `.distributor.json` at the backend root;
- fixed destinations select only the newest discovered bundle;
- fixed destination tie-break behavior is deterministic;
- older discovered bundles are not planned or written to fixed destinations;
- newer source replaces older managed fixed destination state;
- older source skips newer fixed destination state;
- unmanaged non-empty fixed destination fails without force;
- dry-run reports fixed path mapping, candidate count, selected bundle, and
destructive replacement warnings clearly;
- `--force` remains bounded when fixed mapping targets the backend root;
- S3 fixed mapping respects bucket plus prefix as backend root.
## Documentation Updates After Implementation
- Update `docs/config.md` with `path_mapping`.
- Update `docs/operations.md` with archive-plus-latest fan-out examples.
- Update `docs/cli.md` if dry-run output gains fixed-path indicators.
- Add an environment-safe example only after behavior is implemented.
Keep this roadmap under `docs/roadmap/` until implemented.
## Decisions
- `--force` is allowed when fixed mapping targets the backend root, but dry-run
and run output must make the fixed-root replacement explicit and force remains
bounded to the configured backend root.
- Fixed destinations publish only the newest discovered bundle, rather than
processing every discovered bundle and letting later writes replace earlier
writes.
- Dry-run includes an extra warning or summary count for fixed-path destructive
replacements.
## Future Work
- Add richer source selection policies if deployments need something other than
newest-by-`created` for fixed destinations.
- Consider stricter ambiguity handling for equal latest timestamps if real
producer workflows make timestamp ties common.
- Add higher-level status or approval workflows for fixed-root replacements if
dry-run output is not enough operational protection.

View File

@@ -1,179 +0,0 @@
# Roadmap: Link Generation Support
## Purpose
Add destination-aware link generation so distributor can record human-usable URLs
for published artifacts.
This feature should be implemented after destination path mapping so generated
URLs use the final archive or fixed destination path semantics. It prepares the
project for notification adapters, richer inspect/status output, static-site UX,
and future feed generation.
## Current Implementation Grounding
Current publish planning produces `publish.Output` values with destination path,
source path, kind, transform, SHA-256, size, and optional generated data.
Current `.distributor.json` output records contain path, kind, source path,
transform, SHA-256, and size. They do not contain URL metadata, and destination
state has no top-level links block.
Current destination bundle paths preserve the source-root-relative bundle path.
Latest/fixed path mapping is not implemented yet.
## Goals
- Add optional destination-level link configuration.
- Generate per-output URLs when a destination has `links.base_url`.
- Record generated URLs in destination state when configured.
- Select a deterministic primary URL when possible.
- Keep link generation destination-local and independent of producer manifests.
- Do not infer public URLs from backend configuration automatically.
## Non-Goals
- Do not deliver notifications.
- Do not require every destination to expose URLs.
- Do not add static-site indexes or feeds.
- Do not add latest path mapping.
- Do not change source manifest schema.
## Configuration Shape
Add optional destination-level config:
```yaml
links:
base_url: https://weather.example.com
primary: auto
```
Semantics:
- `links.base_url`: absolute HTTP or HTTPS URL corresponding to the destination
backend root.
- `links.primary`: optional primary-link selection policy.
Initial `primary` policies:
- `auto`: choose the best available output;
- `html`: prefer generated HTML outputs;
- `source`: prefer copied source outputs.
If `links` is absent, no URL metadata is generated.
## URL Construction
Construct URLs from:
1. `links.base_url`;
2. destination bundle path relative to the destination backend root;
3. output path relative to the destination bundle path.
Use URL path joining and escaping rules, not filesystem or storage path joins.
Preserve any path prefix in `base_url`. Reject `base_url` values with query
strings or fragments.
For `index.html`, generate a directory-style URL by omitting the filename:
```text
https://weather.example.com/daily/brentwood/2026-06-01/
```
For other outputs, include the output filename:
```text
https://weather.example.com/daily/brentwood/2026-06-01/report.html
```
`index.html` recognition should apply to any output path ending in
`/index.html` or exactly `index.html`.
## Primary Link Selection
For `primary: auto`, select in deterministic publish-plan order:
1. any `index.html` output;
2. first generated HTML output;
3. first source output;
4. no primary URL.
For `primary: html`, prefer generated HTML outputs. For `primary: source`,
prefer source outputs. If no output matches the policy, record no primary link
rather than failing publication.
## Destination State
When links are configured, destination state should record:
- optional per-output URL metadata;
- optional top-level primary URL metadata.
Destinations without `links` should continue producing state without URL
metadata.
Because distributor is still pre-release, URL metadata should be added as
optional state schema version 1 fields rather than introducing destination state
schema version 2 for this feature. Post-release, materially richer state
semantics should use an explicit schema versioning policy.
## Relationship To Other Roadmaps
HTML index mode is not required, but link generation should treat `index.html`
as the preferred URL shape when present.
Latest path destinations should provide the destination bundle path mapping
needed to build fixed/latest URLs correctly.
Producer-facing manifest creation and remote validation are independent of URL
metadata.
CLI display of generated primary links should follow
`docs/roadmap/cli_output_policy.md` so link output is exposed through the shared
`--format text|json` model rather than one-off command output.
## Testing Expectations
Suggested coverage:
- config validation accepts absent `links`;
- config validation accepts valid HTTP and HTTPS base URLs;
- config validation rejects invalid schemes, query strings, and fragments;
- nested bundle paths generate correct URLs;
- `index.html` URLs omit the filename;
- non-index URLs include filenames;
- primary selection is deterministic;
- state records URLs when configured;
- destinations without `links` produce no URL metadata.
## Documentation Updates After Implementation
- Update `docs/config.md` with destination `links` fields.
- Update `docs/operations.md` with static-site URL examples.
- Update `docs/internal/state.md` if destination state changes.
- Update `docs/cli.md` only if run or inspect output displays links.
Keep this roadmap under `docs/roadmap/` until implemented.
## Decisions
- URL metadata is added as optional destination state schema version 1 fields
while distributor remains pre-release.
- v1 implements `links.primary` so future notification and operator-output
features have one canonical primary-link selection policy.
- `run` and `inspect` output should not display generated primary links
immediately. Link display should wait for
`docs/roadmap/cli_output_policy.md` or another explicit inspect/status output
mode.
## Future Work
- Define a post-release destination state schema bump policy before introducing
materially incompatible state changes.
- Add CLI display of primary links through an explicit output mode or status
command, preferably the shared `--format text|json` policy in
`docs/roadmap/cli_output_policy.md`, rather than changing routine `run`
output opportunistically.
- Add richer link policies only if `auto`, `html`, and `source` prove
insufficient.

View File

@@ -1,173 +0,0 @@
# Roadmap: `distributor manifest create`
## Purpose
Add a producer-facing CLI command that creates a valid source `manifest.json` for
a local bundle directory.
This command depends on the public `pkg/bundle` package. It should be useful
for shell scripts and non-Go producers while sharing behavior with Go producers
through the public package.
## Current Implementation Grounding
The current CLI has top-level `version`, `run`, `validate`, and `inspect`
commands. `validate` and `inspect` currently accept local paths only.
The manifest contract is implemented in `internal/bundle`: source manifests have
`schema_version`, `id`, `digest`, `created`, and ordered `files[]` entries with
`path`, `sha256`, and `size`. Validation already enforces path safety,
reserved distributor metadata paths, digest format, duplicate file paths, file
sizes, per-file SHA-256, and the canonical bundle digest.
This command should not add a second implementation of those rules. Once
`pkg/bundle` exists, `manifest create` should call it.
## Goals
- Add a CLI command that writes `manifest.json` for a local bundle directory.
- Reuse public producer-side manifest creation and validation behavior.
- Produce deterministic manifests.
- Preserve caller-provided file order when explicit files are provided.
- Offer convenient recursive scanning when explicit files are omitted.
- Refuse to overwrite an existing manifest unless explicitly requested.
## Non-Goals
- Do not publish or distribute bundles.
- Do not write destination `.distributor.json` state.
- Do not add config-file dependencies.
- Do not add domain-specific metadata.
- Do not expose publish, storage, transform, or destination internals.
## CLI Shape
Proposed syntax:
```sh
distributor manifest create <bundle-path> --id <bundle-id>
```
Optional flags:
```sh
--file <path> Include a bundle-relative file; repeatable.
--created <time> RFC3339 source created timestamp.
--overwrite Replace an existing manifest.json.
```
The first implementation should always write `manifest.json` in the selected
bundle directory. It should not include a stdout/no-write mode or JSON summary
mode until `docs/roadmap/cli_output_policy.md` is implemented or selected for
implementation.
Examples:
```sh
distributor manifest create ./bundle --id weather.daily.2026-06-01
distributor manifest create ./bundle \
--id weather.daily.2026-06-01 \
--created 2026-06-01T11:00:00Z \
--file report.md \
--file summary.txt
```
## File Selection
Use explicit-first behavior:
- if one or more `--file` flags are provided, use exactly those files in flag
order;
- if no `--file` flags are provided, scan the bundle directory recursively and
sort files lexically by slash-separated relative path.
Scanning should include ordinary regular files, including dotfiles, except:
- `manifest.json`;
- `.distributor.json`;
- directories;
- symlinks and other non-regular entries.
Explicit file paths must be relative, clean, slash-separated or normalized to
slash-separated paths, confined to the bundle root, and listed files must be
regular files. Symlinks should be rejected to match current source validation.
## Manifest Creation Behavior
The command should:
1. resolve the local bundle root;
2. determine the selected file list;
3. build file records with SHA-256 and size;
4. compute the canonical bundle digest;
5. set `schema_version` to the current source manifest version;
6. set `id` from `--id`;
7. set `created` from `--created` or the package's selected default behavior;
8. refuse to replace `manifest.json` unless `--overwrite` is set;
9. write `manifest.json`, using temp-and-rename replacement where practical;
10. reload or validate the generated manifest before reporting success.
Success output should be concise:
```text
created manifest.json
bundle: weather.daily.2026-06-01
files: 2
digest: sha256:...
```
## Relationship To Other Roadmaps
`pkg/bundle` is the preferred underlying implementation. If `manifest create`
is implemented first, its reusable manifest-building logic should be structured
so it can move into `pkg/bundle` without changing command behavior.
Remote `validate` and `inspect` should validate bundles after creation but do
not need to participate in manifest writing.
HTML index mode, link generation, and latest paths are publication features and
must not affect source manifest generation.
JSON summary output should follow `docs/roadmap/cli_output_policy.md` rather
than introducing a command-specific `--json` flag or output envelope.
## Testing Expectations
Suggested coverage:
- creates a manifest for a simple local bundle;
- preserves explicit `--file` order;
- scan mode sorts files deterministically;
- scan mode includes dotfiles and excludes distributor metadata files;
- rejects symlinks, unsafe paths, missing explicit files, and non-regular files;
- refuses overwrite without `--overwrite`;
- overwrites only when `--overwrite` is set;
- generated manifests validate with distributor validation;
- CLI help and argument validation match existing CLI style.
## Documentation Updates After Implementation
- Update `docs/cli.md` with command syntax and examples.
- Update `docs/operations.md` with a producer workflow.
- Cross-reference `pkg/bundle` for Go producers once available.
- Add examples only if they are maintained and load/test friendly.
Keep this roadmap under `docs/roadmap/` until implemented.
## Decisions
- v1 does not include `--stdout` or no-write behavior. The command creates or
replaces the bundle's `manifest.json`.
- `--overwrite` is required to replace an existing manifest and should use
temp-and-rename writes where practical.
- v1 does not include JSON summary output. Human-oriented success output remains
consistent with the current CLI unless this work is implemented together with
`docs/roadmap/cli_output_policy.md`.
## Future Work
- Add `--stdout` or another no-write mode if producer pipelines need to capture
manifest JSON directly.
- Add JSON output through the CLI-wide `--format text|json` policy in
`docs/roadmap/cli_output_policy.md`.

View File

@@ -1,284 +0,0 @@
# Roadmap: Public Bundle Manifest Package
## Purpose
Expose a small public Go package that producer applications can import to create
valid distributor source bundle manifests.
The package should encode the producer-side source bundle contract without
exposing distributor's publication, storage, transform, destination state,
notification, or config internals.
## Current Implementation Grounding
The current implementation keeps source bundle behavior in `internal/bundle`:
- `Manifest` and `ManifestFile` model `manifest.json`;
- `ParseManifest` parses JSON and RFC3339 `created` timestamps;
- `ValidateManifest` owns schema version, digest format, file path, duplicate,
size, and bundle digest validation;
- `FileDigest`, `BundleDigest`, and `CanonicalFilePayload` define digest
behavior;
- source validation rejects unsafe paths, reserved distributor metadata paths,
non-file entries, size mismatches, SHA-256 mismatches, and bundle digest
mismatches.
Producers cannot import `internal/bundle`, so Go producers currently need to
duplicate this contract or shell out to future CLI tooling.
## Goals
- Provide a stable producer-facing Go API for manifest creation and validation.
- Reuse the same source manifest, digest, path safety, and RFC3339 behavior used
by distributor validation.
- Keep the public API intentionally small and producer-only.
- Include a safe bundle writer so producers can create complete local bundle
directories without hand-rolling manifest-write and staging behavior.
- Make the future `distributor manifest create` command a thin wrapper over this
package.
- Avoid exposing destination state, publish planning, storage backends,
transforms, notifications, or config.
## Non-Goals
- Do not expose the distributor runner or publication workflow as public API.
- Do not expose storage backends or destination `.distributor.json` state.
- Do not add domain-specific manifest metadata.
- Do not require non-Go producers to use Go APIs.
- Do not implement latest paths, link generation, transforms, or notification
behavior in this package.
## Package Boundary
Use `pkg/bundle` as the public package name.
The package should own only producer-side source bundle concerns:
- manifest model and schema version constant;
- file digest and bundle digest calculation;
- manifest building from producer files;
- manifest JSON load/write helpers;
- manifest and bundle validation;
- source path safety matching distributor validation.
Prefer options structs over long positional functions so future additive
behavior can be introduced without avoidable API churn.
## Initial Exported API
The initial `pkg/bundle` API is locked to the following exported constants,
types, and functions. Implementation should not rename, remove, or reshape
these public symbols during the initial implementation pass.
```go
const ManifestName = "manifest.json"
const SchemaVersion = 1
type Manifest struct {
SchemaVersion int `json:"schema_version"`
ID string `json:"id"`
Digest string `json:"digest"`
Created time.Time `json:"created"`
Files []ManifestFile `json:"files"`
}
type ManifestFile struct {
Path string `json:"path"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
type BuildOptions struct {
Root string
ID string
Created time.Time
Files []string
Scan bool
}
type WriteManifestOptions struct {
Overwrite bool
}
type BundleFile struct {
SourcePath string
Path string
}
type WriteBundleOptions struct {
Root string
ID string
Created time.Time
Files []BundleFile
Overwrite bool
}
func ParseManifest(data []byte) (Manifest, error)
func MarshalManifest(manifest Manifest) ([]byte, error)
func LoadManifest(root string) (Manifest, error)
func WriteManifest(root string, manifest Manifest, opts WriteManifestOptions) error
func BuildManifest(opts BuildOptions) (Manifest, error)
func ValidateManifest(manifest Manifest) error
func ValidateBundle(root string, manifest Manifest) error
func WriteBundle(opts WriteBundleOptions) (Manifest, error)
func ValidateSourcePath(path string) error
func FileDigest(data []byte) string
func BundleDigest(files []ManifestFile) string
func CanonicalFilePayload(files []ManifestFile) string
```
## API Semantics
`BuildManifest` requires `Root`, `ID`, and exactly one file-selection mode:
explicit `Files` or `Scan: true`.
Explicit `Files` preserve caller order. `Scan: true` recursively scans `Root`,
includes regular files including dotfiles, excludes `manifest.json` and
`.distributor.json`, rejects symlinks, and sorts by slash-separated relative
path.
Zero `Created` values default to the current UTC time. All public path fields
use slash-separated bundle-relative paths.
`MarshalManifest` validates before marshaling and emits deterministic JSON with
fixed field order and a trailing newline.
`WriteManifest` writes `manifest.json`; it fails if the file exists unless
`WriteManifestOptions.Overwrite` is true, and it uses temp-and-rename
replacement where practical.
`ValidateManifest` checks manifest-only semantics. `ValidateBundle` checks the
supplied manifest against local files under `root`, including existence,
regular-file type, size, SHA-256, path safety, duplicates, and bundle digest.
`WriteBundle` copies existing local files from `BundleFile.SourcePath` into a
staged bundle at `BundleFile.Path`, writes a compliant manifest, validates the
staged bundle, and promotes it to `WriteBundleOptions.Root`.
`WriteBundleOptions.Overwrite` permits replacing an existing bundle root.
Replacement must build the new bundle completely before touching the existing
root, then use sibling temp and backup paths for best-effort promotion and
restore on failure.
The writer remains producer-side and filesystem-local. It must not expose
distributor storage backends or publication behavior.
## Manifest Compatibility
The package should treat the source manifest schema as a compatibility boundary:
- export the current schema version;
- preserve JSON field names exactly;
- use lowercase `sha256:<64 hex>` digests;
- use slash-separated relative paths in JSON;
- use RFC3339 timestamps;
- default a zero build or writer `Created` value to the current UTC time;
- preserve caller-provided file order for explicit file lists;
- produce deterministic ordering when scan-based building is selected;
- reject symlinks if distributor validation still rejects source symlinks.
Scan-based building belongs in v1 of the public package. Explicit file lists
should preserve caller order. Scan mode should sort by slash-separated relative
path and share the same filtering rules expected by future CLI manifest
creation.
The internal implementation may either move source-bundle core logic into
`pkg/bundle` and have internal packages consume it, or keep internal wrappers
around public core logic. The important invariant is that public package,
future CLI manifest creation, and distributor validation must not drift.
## Relationship To Other Roadmaps
`distributor manifest create` should call `pkg/bundle` rather than maintaining a
separate manifest builder.
Remote `validate` and `inspect` should continue using distributor's storage
abstraction and internal app wiring; they do not need public producer APIs.
HTML index mode, link generation, and latest path destinations operate after a
bundle has already entered distributor and should not affect this package.
## Testing Expectations
Suggested coverage:
- build a manifest from explicit files;
- build a manifest by scanning a local bundle root;
- preserve explicit file order;
- sort scan results deterministically by slash-separated relative path;
- compute per-file SHA-256 and size;
- compute the expected canonical bundle digest;
- write and load `manifest.json`;
- default zero `Created` to current UTC time while honoring explicit timestamps;
- validate generated manifests successfully;
- reject unsafe paths, missing files, non-regular files, and symlinks;
- emit slash-separated JSON paths;
- parse, marshal, load, and write manifests through the exact exported API;
- fail `WriteManifest` when `manifest.json` exists unless overwrite is enabled;
- emit deterministic manifest JSON with fixed field order and trailing newline;
- write a complete local bundle through the public writer;
- copy `BundleFile.SourcePath` content to the configured bundle-relative path;
- support `WriteBundleOptions.Overwrite` through staged replacement;
- avoid leaving a completed bundle path without a valid manifest when writer
staging or promotion fails where practical;
- compile public examples under `go test` where practical;
- prove consistency with distributor validation fixtures.
## Documentation Updates After Implementation
- Add Go package documentation under `pkg/bundle`.
- Update `README.md` to mention Go producer support.
- Update `docs/operations.md` with a producer integration example.
- Cross-reference `distributor manifest create` once that CLI command exists.
Keep this roadmap under `docs/roadmap/` until implemented.
## Implementation Stages
`docs/roadmap/implementation.md` intentionally splits this roadmap across two
implementation prompts: package core/building first, then the local bundle
writer. The split keeps the public API extraction separate from producer-side
bundle assembly.
Core/building stage:
1. Move or wrap the existing source manifest model, digest logic, path
validation, and RFC3339 handling so `pkg/bundle` and internal validation use
one contract.
2. Implement the locked Stage 2 public API symbols:
`ManifestName`, `SchemaVersion`, `Manifest`, `ManifestFile`,
`BuildOptions`, `WriteManifestOptions`, `ParseManifest`,
`MarshalManifest`, `LoadManifest`, `WriteManifest`, `BuildManifest`,
`ValidateManifest`, `ValidateBundle`, `ValidateSourcePath`, `FileDigest`,
`BundleDigest`, and `CanonicalFilePayload`.
3. Add explicit-list and scan-based manifest building APIs, including zero
`Created` defaulting to current UTC time.
4. Update internal packages to consume the shared implementation without
changing current validation behavior.
Writer stage:
1. Implement the locked Stage 3 public API symbols: `BundleFile`,
`WriteBundleOptions`, and `WriteBundle`.
2. Add the local bundle writer with staged promotion, atomic filesystem
operations where practical, and overwrite behavior through sibling temp and
backup paths.
3. Add package documentation and producer-facing examples.
## Decisions
- A zero `Created` value defaults to the current UTC time. Producers may still
provide explicit timestamps for reproducible or backfilled bundles.
- Scan-based manifest building is included in v1, behind explicit options.
Explicit file lists preserve caller order; scan mode sorts deterministically.
- A local bundle writer is included in v1. It should be safe and producer-side,
but it must not expose distributor publication or storage internals.
- The exported API names and signatures in `Initial Exported API` are
normative for implementation.
## Future Work
- Broader producer workflow helpers, such as richer ignore rules or template
scaffolding, can be considered after the first public package exists.
- Remote or storage-backed producer writers remain out of scope unless a future
producer use case requires them.

View File

@@ -1,158 +0,0 @@
# Roadmap: Remote `validate` and `inspect`
## Purpose
Expand `distributor validate` and `distributor inspect` so they can operate on
configured pipeline sources, including `local`, `ssh`, and `s3` sources.
This feature improves operator diagnostics after producer-side manifest
creation is available.
## Current Implementation Grounding
Current `validate` and `inspect` accept one local path. App-level code opens a
local backend with `openLocalPath`, then uses `bundle.Discover`, which walks
storage, finds `manifest.json`, and validates bundles.
`run` already has the app-level backend factory for configured local, SSH, and
S3 sources. `run` also loads `secrets.directory` before backend construction so
explicit S3 credential environment references can resolve without mutating the
process environment.
This feature should reuse that source backend construction path for configured
sources and remain read-only.
## Goals
- Preserve the existing local path shortcut:
`distributor validate <path>` and `distributor inspect <path>`.
- Add config-driven source validation and inspection:
`--config <path> --pipeline <id>`.
- Support configured `local`, `ssh`, and `s3` sources.
- Reuse the same source discovery and validation behavior used by `run`.
- Reuse secrets loading and backend credential resolution from `run`.
- Avoid opening or inspecting destinations in the first version.
- Keep output concise and operator-oriented.
## Non-Goals
- Do not write to source or destination storage.
- Do not inspect destination state in v1.
- Do not add new backend types.
- Do not change source manifest schema.
- Do not add daemon, API, or producer-package dependencies.
## CLI Mode Rules
Use two mutually exclusive modes:
```sh
distributor validate <local-path>
distributor inspect <local-path>
```
and:
```sh
distributor validate --config config.yml --pipeline weather-daily
distributor inspect --config config.yml --pipeline weather-daily
```
Do not allow positional local paths together with `--config` or `--pipeline`.
Require `--pipeline` in config mode, even when the config has exactly one
pipeline. This avoids surprising remote access and keeps the initial behavior
explicit.
Optional narrowing:
```sh
--bundle <source-root-relative-bundle-path>
```
When provided, `--bundle` identifies a source-root-relative bundle directory and
the command validates or inspects that bundle rather than discovering all
bundles.
## Behavior
For config mode:
1. load config using the same defaulting and validation path as `run`;
2. load `secrets.directory`;
3. find the requested pipeline id;
4. open only the pipeline source backend;
5. discover all bundles or validate the requested `--bundle`;
6. validate manifest schema, paths, files, per-file digests, and bundle digest;
7. return non-zero if any selected bundle fails validation.
`inspect` should fully validate selected bundles by default, matching current
`bundle.Discover` behavior. Output can then report reliable normalized metadata:
- pipeline id for config mode;
- backend type;
- source-relative bundle path;
- bundle id;
- created timestamp;
- digest;
- file count and total size;
- file path, size, and SHA-256.
## Relationship To Other Roadmaps
Remote validation works well after `manifest create` and `pkg/bundle` because
operators can validate producer output where it actually lands.
This feature does not depend on HTML index mode, link generation, or latest path
destinations. Those features may later make inspection output richer, but v1
should stay source-focused.
Machine-readable output should follow `docs/roadmap/cli_output_policy.md` and
use the shared `--format text|json` policy rather than command-specific JSON
flags.
## Testing Expectations
Suggested coverage:
- existing local path validation still works;
- existing local path inspection still works;
- config-mode local source validation works;
- fake configured source validation works through the storage abstraction;
- `--bundle` validates a specific source-relative bundle when implemented;
- missing pipeline id fails clearly;
- positional path plus `--config` is rejected;
- source backend open failures include pipeline/backend context;
- digest mismatch and missing manifest failures are clear;
- inspect output includes normalized source metadata.
Avoid live SSH or S3 tests except existing opt-in integration patterns.
## Documentation Updates After Implementation
- Update `docs/cli.md` with both local and config-driven syntax.
- Update `docs/operations.md` with remote validation examples.
- Update `docs/troubleshooting.md` for common configured-source failures.
- Update examples only if useful and environment-gated.
Keep this roadmap under `docs/roadmap/` until implemented.
## Decisions
- Destination-state inspection is not part of v1 remote `validate` or
`inspect`. These commands remain source-focused and should not open
destinations unexpectedly.
- JSON output is deferred to `docs/roadmap/cli_output_policy.md` or to an
implementation pass that selects that policy.
- `--pipeline` remains required in config mode, even when a config has exactly
one pipeline.
## Future Work
- Add destination-state inspection behind an explicit flag such as
`--with-destinations` if operators need fan-out status diagnostics from
`inspect`.
- Add JSON output through the CLI-wide `--format text|json` policy in
`docs/roadmap/cli_output_policy.md`.
- Reconsider optional pipeline selection only if the project later introduces a
broader command mode for single-pipeline configs.

View File

@@ -36,6 +36,67 @@ rg -n "backend:" <config-path>
Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for executable workflows.
## `--format: format must be text or json`
Likely cause: a command was run with an unsupported output format.
Diagnostic:
```sh
go run ./cmd/distributor run --help
```
Safe fix: use `--format text` or `--format json`. Help and usage output are always text.
## `--format json` wrote no JSON output
Likely cause: the command failed before it could construct a result, such as a missing config file, invalid arguments, unreadable secrets directory, or source setup failure.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --format json
```
Safe fix: read the stderr error and fix the setup problem. JSON mode writes a document only after the command has enough information to construct a result.
## `configured source mode requires --pipeline`
Likely cause: `validate` or `inspect` was run with `--config` but without an explicit pipeline id.
Diagnostic:
```sh
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```
Safe fix: add `--pipeline <pipeline-id>`. Configured source diagnostics require an explicit pipeline even when the config contains one pipeline.
## `does not accept a local path with --config, --pipeline, or --bundle`
Likely cause: local-path mode and configured source mode were mixed in one `validate` or `inspect` command.
Diagnostic:
```sh
go run ./cmd/distributor inspect --help
```
Safe fix: use either `distributor inspect <local-path>` or `distributor inspect --config <path> --pipeline <id>`, not both.
## `--format json` exited non-zero with `ok: false`
Likely cause: `run` began planning or executing destinations, and at least one destination failed while other destination results were still available.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --format json
```
Safe fix: inspect the top-level `errors` array, `result.actions`, and `result.summary`. Fix the failed destination, then preview with `--dry-run --format json` before retrying.
## `prefix must be a clean relative slash-separated path`
Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
@@ -198,21 +259,33 @@ ssh-keygen -F <host> -f <known-hosts-path>
Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key.
## `stat ssh ... not_found` or `no bundles found`
## `pipeline "<id>" not found`
Likely cause: the configured SSH `path` is wrong, unreadable, or does not contain source bundles.
Likely cause: configured source validation or inspection requested a pipeline id that is not present in the config file.
Diagnostic:
```sh
sftp <user>@<host>
rg -n "id:" <config-path>
```
Safe fix: correct the remote root `path`, permissions, or source bundle location.
Safe fix: pass an existing pipeline id with `--pipeline`, or update the config.
## `stat ssh ... not_found`, `stat s3 ... not_found`, or `no bundles found`
Likely cause: the configured source root is wrong, unreadable, or does not contain source bundles.
Diagnostic:
```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
```
Safe fix: correct the configured source root, S3 prefix, permissions, or source bundle location. Use `--bundle <path>` only with a source-root-relative bundle directory that contains `manifest.json`.
## `validate command requires a path` or `inspect command requires a path`
Likely cause: `validate` or `inspect` was run without a path.
Likely cause: `validate` or `inspect` was run without a local path and without configured source mode.
Diagnostic:
@@ -221,7 +294,7 @@ go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```
Safe fix: pass a local source bundle directory or a local tree containing source bundles.
Safe fix: pass a local source bundle directory or local tree, or pass both `--config <path>` and `--pipeline <id>`.
## `no bundles found under "."`

View File

@@ -0,0 +1,29 @@
pipelines:
- id: example-archive-and-latest
source:
backend: local
path: examples/source-bundle
destinations:
- id: local-source-archive
backend: local
path: workspace/published/archive-and-latest/archive
path_mapping:
mode: preserve_relative
publish:
source: true
html: false
- id: local-html-latest
backend: local
path: workspace/published/archive-and-latest/latest
path_mapping:
mode: fixed
links:
base_url: https://reports.example.com/latest
primary: auto
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index

16
examples/local-index.yml Normal file
View File

@@ -0,0 +1,16 @@
pipelines:
- id: example-index-bundle
source:
backend: local
path: examples/source-bundle
destinations:
- id: local-index
backend: local
path: workspace/published/index-bundle
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index

View File

@@ -5,38 +5,130 @@ import (
"fmt"
"io"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
type InspectOptions struct {
Path string
Stdout io.Writer
Path string
ConfigPath string
PipelineID string
BundlePath string
Stdout io.Writer
OutputFormat OutputFormat
}
func Inspect(ctx context.Context, options InspectOptions) error {
if options.Path == "" {
return fmt.Errorf("inspect command requires a path")
}
backend, err := newBackendFactory().openLocalPath(ctx, options.Path)
if err != nil {
return err
}
bundles, err := bundle.Discover(ctx, backend, "")
if err != nil {
return err
}
return writeInspection(options.Stdout, bundles)
return inspectWithBackendFactory(ctx, options, newBackendFactoryWithEnvironment)
}
func writeInspection(w io.Writer, bundles []bundle.Bundle) error {
func inspectWithBackendFactory(ctx context.Context, options InspectOptions, provider backendFactoryProvider) error {
if err := ValidateOutputFormat(options.OutputFormat); err != nil {
return err
}
selection, err := selectSourceBundles(ctx, sourceCommandOptions{
CommandName: "inspect",
Path: options.Path,
ConfigPath: options.ConfigPath,
PipelineID: options.PipelineID,
BundlePath: options.BundlePath,
}, provider)
if err != nil {
return err
}
return writeInspectResult(options, selection)
}
func inspectConfigWithBackendFactory(ctx context.Context, cfg config.Config, options InspectOptions, provider backendFactoryProvider) error {
if err := ValidateOutputFormat(options.OutputFormat); err != nil {
return err
}
selection, err := selectSourceBundlesFromConfig(ctx, cfg, sourceCommandOptions{
CommandName: "inspect",
PipelineID: options.PipelineID,
BundlePath: options.BundlePath,
}, provider)
if err != nil {
return err
}
return writeInspectResult(options, selection)
}
func writeInspectResult(options InspectOptions, selection sourceSelection) error {
if IsJSONOutput(options.OutputFormat) {
return WriteJSONEnvelope(options.Stdout, "inspect", true, selection.Warnings, inspectResultFromSelection(selection), nil)
}
if err := writeWarnings(options.Stdout, selection.Warnings); err != nil {
return err
}
return writeInspection(options.Stdout, selection)
}
type inspectResult struct {
PipelineID string `json:"pipeline_id,omitempty"`
SourceBackend string `json:"source_backend,omitempty"`
BundleCount int `json:"bundle_count"`
Bundles []inspectBundleResult `json:"bundles"`
}
type inspectBundleResult struct {
Path string `json:"path"`
ID string `json:"id"`
Created string `json:"created"`
Digest string `json:"digest"`
FileCount int `json:"file_count"`
TotalSize int64 `json:"total_size"`
Files []inspectFileResult `json:"files"`
}
type inspectFileResult struct {
Path string `json:"path"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
func inspectResultFromSelection(selection sourceSelection) inspectResult {
result := inspectResult{
PipelineID: selection.PipelineID,
SourceBackend: selection.SourceBackend,
BundleCount: len(selection.Bundles),
Bundles: make([]inspectBundleResult, 0, len(selection.Bundles)),
}
for _, sourceBundle := range selection.Bundles {
bundleResult := inspectBundleResult{
Path: storage.DisplayPath(sourceBundle.RootRelativePath),
ID: sourceBundle.Manifest.ID,
Created: sourceBundle.Manifest.Created.Format("2006-01-02T15:04:05Z07:00"),
Digest: sourceBundle.Manifest.Digest,
FileCount: len(sourceBundle.Manifest.Files),
Files: make([]inspectFileResult, 0, len(sourceBundle.Manifest.Files)),
}
for _, file := range sourceBundle.Manifest.Files {
bundleResult.TotalSize += file.Size
bundleResult.Files = append(bundleResult.Files, inspectFileResult{
Path: file.Path,
SHA256: file.SHA256,
Size: file.Size,
})
}
result.Bundles = append(result.Bundles, bundleResult)
}
return result
}
func writeInspection(w io.Writer, selection sourceSelection) error {
if w == nil {
return nil
}
if _, err := fmt.Fprintf(w, "Bundles: %d\n", len(bundles)); err != nil {
if selection.ConfigMode {
if _, err := fmt.Fprintf(w, "Pipeline: %s\nSource: %s\n", selection.PipelineID, selection.SourceBackend); err != nil {
return err
}
}
if _, err := fmt.Fprintf(w, "Bundles: %d\n", len(selection.Bundles)); err != nil {
return err
}
for _, sourceBundle := range bundles {
for _, sourceBundle := range selection.Bundles {
if _, err := fmt.Fprintf(
w,
"- path=%s id=%s created=%s digest=%s files=%d\n",

View File

@@ -6,6 +6,8 @@ import (
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
)
func TestInspectPrintsBundleSummary(t *testing.T) {
@@ -32,6 +34,57 @@ func TestInspectPrintsBundleSummary(t *testing.T) {
}
}
func TestInspectConfiguredLocalSource(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "daily", testutil.BundleOptions{ID: "reports.daily"})
var stdout bytes.Buffer
err := Inspect(context.Background(), InspectOptions{
ConfigPath: testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot),
PipelineID: "reports",
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Inspect() configured source error = %v", err)
}
output := stdout.String()
for _, want := range []string{
"Pipeline: reports",
"Source: local",
"Bundles: 1",
"path=daily",
"id=reports.daily",
} {
if !strings.Contains(output, want) {
t.Fatalf("Inspect() output = %q, want substring %q", output, want)
}
}
}
func TestInspectConfiguredSourceJSON(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{ID: "reports.json"})
var stdout bytes.Buffer
err := Inspect(context.Background(), InspectOptions{
ConfigPath: testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot),
PipelineID: "reports",
Stdout: &stdout,
OutputFormat: OutputFormatJSON,
})
if err != nil {
t.Fatalf("Inspect() configured JSON error = %v", err)
}
result := decodeAppResult(t, stdout.String())
if result["pipeline_id"] != "reports" || result["source_backend"] != "local" || result["bundle_count"] != float64(1) {
t.Fatalf("result = %#v, want configured inspect metadata", result)
}
}
func TestInspectRequiresPath(t *testing.T) {
err := Inspect(context.Background(), InspectOptions{})
if err == nil || !strings.Contains(err.Error(), "requires a path") {

128
internal/app/manifest.go Normal file
View File

@@ -0,0 +1,128 @@
package app
import (
"context"
"fmt"
"io"
"path/filepath"
"strings"
"time"
producerbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
)
type ManifestCreateOptions struct {
Root string
ID string
Created string
Files []string
Overwrite bool
Stdout io.Writer
OutputFormat OutputFormat
}
func ManifestCreate(ctx context.Context, options ManifestCreateOptions) error {
if err := ValidateOutputFormat(options.OutputFormat); err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
if options.Root == "" {
return fmt.Errorf("manifest create command requires a bundle path")
}
if options.ID == "" {
return fmt.Errorf("manifest create command requires --id")
}
created, err := parseOptionalCreated(options.Created)
if err != nil {
return err
}
files := normalizeManifestFiles(options.Files)
buildOptions := producerbundle.BuildOptions{
Root: options.Root,
ID: options.ID,
Created: created,
Files: files,
Scan: len(files) == 0,
}
manifest, err := producerbundle.BuildManifest(buildOptions)
if err != nil {
return err
}
if err := producerbundle.WriteManifest(options.Root, manifest, producerbundle.WriteManifestOptions{Overwrite: options.Overwrite}); err != nil {
return err
}
loaded, err := producerbundle.LoadManifest(options.Root)
if err != nil {
return err
}
if err := producerbundle.ValidateBundle(options.Root, loaded); err != nil {
return err
}
result := manifestCreateResultFromManifest(options.Root, loaded)
if IsJSONOutput(options.OutputFormat) {
return WriteJSONEnvelope(options.Stdout, "manifest create", true, nil, result, nil)
}
if options.Stdout != nil {
_, err = fmt.Fprintf(options.Stdout, "created %s\nbundle: %s\nfiles: %d\ndigest: %s\n", producerbundle.ManifestName, result.ID, result.FileCount, result.Digest)
}
return err
}
func parseOptionalCreated(value string) (time.Time, error) {
if value == "" {
return time.Time{}, nil
}
created, err := time.Parse(time.RFC3339, value)
if err != nil {
return time.Time{}, fmt.Errorf("created must be RFC3339: %w", err)
}
return created, nil
}
func normalizeManifestFiles(files []string) []string {
normalized := make([]string, 0, len(files))
for _, file := range files {
normalized = append(normalized, filepath.ToSlash(filepath.Clean(strings.ReplaceAll(file, "\\", string(filepath.Separator)))))
}
return normalized
}
type manifestCreateResult struct {
ManifestPath string `json:"manifest_path"`
Root string `json:"root"`
ID string `json:"id"`
Created string `json:"created"`
Digest string `json:"digest"`
FileCount int `json:"file_count"`
Files []manifestCreateFileResult `json:"files"`
}
type manifestCreateFileResult struct {
Path string `json:"path"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
func manifestCreateResultFromManifest(root string, manifest producerbundle.Manifest) manifestCreateResult {
result := manifestCreateResult{
ManifestPath: filepath.ToSlash(filepath.Join(root, producerbundle.ManifestName)),
Root: filepath.ToSlash(root),
ID: manifest.ID,
Created: manifest.Created.Format(time.RFC3339),
Digest: manifest.Digest,
FileCount: len(manifest.Files),
Files: make([]manifestCreateFileResult, 0, len(manifest.Files)),
}
for _, file := range manifest.Files {
result.Files = append(result.Files, manifestCreateFileResult{
Path: file.Path,
SHA256: file.SHA256,
Size: file.Size,
})
}
return result
}

76
internal/app/output.go Normal file
View File

@@ -0,0 +1,76 @@
package app
import (
"encoding/json"
"fmt"
"io"
)
const outputSchemaVersion = 1
type OutputFormat string
const (
OutputFormatText OutputFormat = "text"
OutputFormatJSON OutputFormat = "json"
)
type OutputWarning struct {
Message string `json:"message"`
}
type OutputError struct {
PipelineID string `json:"pipeline_id,omitempty"`
DestinationID string `json:"destination_id,omitempty"`
Backend string `json:"backend,omitempty"`
BundlePath string `json:"bundle_path,omitempty"`
Message string `json:"message"`
}
type outputEnvelope struct {
SchemaVersion int `json:"schema_version"`
Command string `json:"command"`
OK bool `json:"ok"`
Warnings []OutputWarning `json:"warnings"`
Result any `json:"result"`
Errors []OutputError `json:"errors,omitempty"`
}
func NormalizeOutputFormat(format OutputFormat) OutputFormat {
if format == "" {
return OutputFormatText
}
return format
}
func ValidateOutputFormat(format OutputFormat) error {
switch NormalizeOutputFormat(format) {
case OutputFormatText, OutputFormatJSON:
return nil
default:
return fmt.Errorf("format must be text or json")
}
}
func IsJSONOutput(format OutputFormat) bool {
return NormalizeOutputFormat(format) == OutputFormatJSON
}
func WriteJSONEnvelope(w io.Writer, command string, ok bool, warnings []OutputWarning, result any, errors []OutputError) error {
if w == nil {
return nil
}
if warnings == nil {
warnings = []OutputWarning{}
}
envelope := outputEnvelope{
SchemaVersion: outputSchemaVersion,
Command: command,
OK: ok,
Warnings: warnings,
Result: result,
Errors: errors,
}
encoder := json.NewEncoder(w)
return encoder.Encode(envelope)
}

View File

@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"io"
"sort"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
@@ -15,14 +16,18 @@ import (
)
type RunOptions struct {
ConfigPath string
DryRun bool
Force bool
Stdout io.Writer
Notifier notify.Notifier
ConfigPath string
DryRun bool
Force bool
Stdout io.Writer
OutputFormat OutputFormat
Notifier notify.Notifier
}
func Run(ctx context.Context, options RunOptions) error {
if err := ValidateOutputFormat(options.OutputFormat); err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
@@ -49,28 +54,41 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
if notifier == nil {
notifier = notify.Noop{}
}
jsonOutput := IsJSONOutput(options.OutputFormat)
summary := runSummary{dryRun: options.DryRun}
result := runResult{
DryRun: options.DryRun,
Pipelines: []runPipelineResult{},
Actions: []runActionResult{},
}
var warnings []OutputWarning
var failures runFailures
secretLoad, err := config.LoadSecretEnvironment(cfg.Secrets.Directory, nil)
if err != nil {
return err
}
if options.Stdout != nil {
if err := writeSecretConflictWarnings(options.Stdout, secretLoad.Conflicts); err != nil {
secretWarnings := secretConflictWarnings(secretLoad.Conflicts)
if jsonOutput {
warnings = append(warnings, secretWarnings...)
} else if options.Stdout != nil {
if err := writeWarnings(options.Stdout, secretWarnings); err != nil {
return err
}
}
backends := provider(secretLoad.Environment)
backends.readOnlyKnownHosts = options.DryRun
transforms := newTransformRegistry()
if options.Stdout != nil {
if options.Stdout != nil && !jsonOutput {
if _, err := fmt.Fprintf(options.Stdout, "Configured pipelines: %d\n", len(cfg.Pipelines)); err != nil {
return err
}
}
for _, pipeline := range cfg.Pipelines {
if options.Stdout != nil {
if err := writeSSHWarnings(options.Stdout, pipeline); err != nil {
pipelineWarnings := sshWarnings(pipeline)
if jsonOutput {
warnings = append(warnings, pipelineWarnings...)
} else if options.Stdout != nil {
if err := writeWarnings(options.Stdout, pipelineWarnings); err != nil {
return err
}
}
@@ -83,53 +101,114 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
closeBackend(sourceBackend)
return fmt.Errorf("pipeline %s source backend %s discover source bundles: %w", pipeline.ID, pipeline.Source.Backend, err)
}
if options.Stdout != nil {
result.Pipelines = append(result.Pipelines, runPipelineResult{
ID: pipeline.ID,
SourceBackend: pipeline.Source.Backend,
BundleCount: len(bundles),
Destinations: destinationIDs(pipeline.Destinations),
})
if options.Stdout != nil && !jsonOutput {
if _, err := fmt.Fprintf(options.Stdout, "- pipeline=%s source=%s bundles=%d destinations=%s\n", pipeline.ID, pipeline.Source.Backend, len(bundles), destinationSummary(pipeline.Destinations)); err != nil {
closeBackend(sourceBackend)
return err
}
}
for _, sourceBundle := range bundles {
for _, destination := range pipeline.Destinations {
destinationBackend, err := backends.openDestination(ctx, destination)
if err != nil {
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
for _, destination := range pipeline.Destinations {
selections := selectDestinationBundles(destination, bundles)
if isFixedPathDestination(destination) {
summary.recordFixedPath()
if options.DryRun {
warning := fixedPathSelectionWarning(pipeline.ID, destination.ID, selections, len(bundles))
if jsonOutput {
warnings = append(warnings, warning)
} else if options.Stdout != nil {
if err := writeWarnings(options.Stdout, []OutputWarning{warning}); err != nil {
closeBackend(sourceBackend)
return err
}
}
}
}
if len(selections) == 0 {
continue
}
destinationBackend, err := backends.openDestination(ctx, destination)
if err != nil {
for _, selection := range selections {
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(selection.SourceBundle.RootRelativePath), err)
summary.recordFailure()
if options.Stdout != nil {
writeErrorLine(options.Stdout, sourceBundle.RootRelativePath, destination.ID, destination.Backend, err)
}
continue
}
closeDestination := true
deferCloseDestination := func() {
if closeDestination {
closeBackend(destinationBackend)
closeDestination = false
if jsonOutput {
result.Actions = append(result.Actions, errorAction(pipeline.ID, destination.ID, destination.Backend, selection.SourceBundle.RootRelativePath, err))
} else if options.Stdout != nil {
writeErrorLine(options.Stdout, selection.SourceBundle.RootRelativePath, destination.ID, destination.Backend, err)
}
}
continue
}
closeDestination := true
deferCloseDestination := func() {
if closeDestination {
closeBackend(destinationBackend)
closeDestination = false
}
}
for _, selection := range selections {
sourceBundle := selection.SourceBundle
req := publish.Request{
PipelineID: pipeline.ID,
DestinationID: destination.ID,
SourceBundle: sourceBundle,
SourceBackend: sourceBackend,
DestinationBackend: destinationBackend,
DestinationBundlePath: sourceBundle.RootRelativePath,
DestinationBundlePath: selection.DestinationBundlePath,
PathMapping: destination.PathMap.Mode,
Publish: *destination.Publish,
Transform: destination.Transform,
Links: destination.Links,
Transformers: transforms,
Transfer: destination.Transfer,
DistributorVersion: Version,
Force: options.Force,
}
plan, err := publish.Build(ctx, req)
if err != nil && plan.DestinationID == "" {
plan = publish.Plan{DestinationID: destination.ID, BundlePath: sourceBundle.RootRelativePath}
if err != nil {
if plan.PipelineID == "" {
plan.PipelineID = pipeline.ID
}
if plan.DestinationID == "" {
plan.DestinationID = destination.ID
}
if plan.BundleID == "" {
plan.BundleID = sourceBundle.Manifest.ID
}
if plan.BundlePath == "" {
plan.BundlePath = sourceBundle.RootRelativePath
}
if plan.DestinationBundlePath == "" {
plan.DestinationBundlePath = selection.DestinationBundlePath
}
}
if options.Stdout != nil {
if isFixedPathDestination(destination) {
plan.PathMapping = config.PathMappingFixed
if options.DryRun && isDestructiveFixedPathAction(plan.Action) {
warning := fixedPathReplacementWarning(plan)
if jsonOutput {
warnings = append(warnings, warning)
} else if options.Stdout != nil {
if err := writeWarnings(options.Stdout, []OutputWarning{warning}); err != nil {
deferCloseDestination()
closeBackend(sourceBackend)
return err
}
}
}
}
if jsonOutput {
result.Actions = append(result.Actions, runActionFromPlan(destination.Backend, plan, err))
} else if options.Stdout != nil {
writePlanLine(options.Stdout, destination.Backend, plan, err)
}
if err != nil {
deferCloseDestination()
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
summary.recordFailure()
continue
@@ -137,26 +216,29 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
summary.recordPlan(plan.Action)
if !options.DryRun {
if err := publish.Execute(ctx, req, plan); err != nil {
deferCloseDestination()
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
summary.recordFailure()
continue
}
if shouldNotify(plan.Action) {
if err := notifier.Notify(ctx, notifyEvent(plan)); err != nil {
deferCloseDestination()
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
summary.recordFailure()
continue
}
}
}
deferCloseDestination()
}
deferCloseDestination()
}
closeBackend(sourceBackend)
}
if options.Stdout != nil {
result.Summary = summary.Result()
if jsonOutput {
if err := WriteJSONEnvelope(options.Stdout, "run", len(failures.items) == 0, warnings, result, failures.outputErrors()); err != nil {
return err
}
} else if options.Stdout != nil {
if _, err := fmt.Fprintln(options.Stdout, summary.Line()); err != nil {
return err
}
@@ -188,10 +270,17 @@ func writePlanLine(w io.Writer, backend string, plan publish.Plan, planErr error
if destinationID == "" {
destinationID = "unknown"
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s action=error reason=%q\n", storage.DisplayPath(plan.BundlePath), destinationID, backend, planErr.Error())
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=error reason=%q\n", storage.DisplayPath(plan.BundlePath), destinationID, backend, pathMappingSummary(plan), planErr.Error())
return
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s action=%s outputs=%s reason=%q\n", storage.DisplayPath(plan.BundlePath), plan.DestinationID, backend, plan.Action, outputSummary(plan.Outputs), plan.Reason)
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=%s outputs=%s reason=%q\n", storage.DisplayPath(plan.BundlePath), plan.DestinationID, backend, pathMappingSummary(plan), plan.Action, outputSummary(plan.Outputs), plan.Reason)
}
func pathMappingSummary(plan publish.Plan) string {
if plan.PathMapping != config.PathMappingFixed {
return ""
}
return fmt.Sprintf(" path_mapping=fixed target=%s", storage.DisplayPath(plan.DestinationBundlePath))
}
func writeErrorLine(w io.Writer, bundlePath, destinationID, backend string, err error) {
@@ -212,6 +301,74 @@ func outputSummary(outputs []publish.Output) string {
return strings.Join(paths, ",")
}
type destinationBundleSelection struct {
SourceBundle bundle.Bundle
DestinationBundlePath string
}
func selectDestinationBundles(destination config.Destination, bundles []bundle.Bundle) []destinationBundleSelection {
if !isFixedPathDestination(destination) {
selections := make([]destinationBundleSelection, 0, len(bundles))
for _, sourceBundle := range bundles {
selections = append(selections, destinationBundleSelection{
SourceBundle: sourceBundle,
DestinationBundlePath: sourceBundle.RootRelativePath,
})
}
return selections
}
if len(bundles) == 0 {
return nil
}
sourceBundle := newestBundle(bundles)
return []destinationBundleSelection{{
SourceBundle: sourceBundle,
DestinationBundlePath: "",
}}
}
func newestBundle(bundles []bundle.Bundle) bundle.Bundle {
if len(bundles) == 0 {
return bundle.Bundle{}
}
sorted := append([]bundle.Bundle(nil), bundles...)
sort.Slice(sorted, func(i, j int) bool {
if sorted[i].Manifest.Created.Equal(sorted[j].Manifest.Created) {
return sorted[i].RootRelativePath < sorted[j].RootRelativePath
}
return sorted[i].Manifest.Created.After(sorted[j].Manifest.Created)
})
return sorted[0]
}
func isFixedPathDestination(destination config.Destination) bool {
return destination.PathMap.Mode == config.PathMappingFixed
}
func fixedPathSelectionWarning(pipelineID, destinationID string, selections []destinationBundleSelection, candidateCount int) OutputWarning {
selected := "none"
if len(selections) > 0 {
selected = storage.DisplayPath(selections[0].SourceBundle.RootRelativePath)
}
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed candidates=%d selected_bundle=%s destination_bundle=.", pipelineID, destinationID, candidateCount, selected)}
}
func isDestructiveFixedPathAction(action publish.Action) bool {
return action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
}
func fixedPathReplacementWarning(plan publish.Plan) OutputWarning {
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed action=%s replaces destination root for selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Action, storage.DisplayPath(plan.BundlePath))}
}
func destinationIDs(destinations []config.Destination) []string {
ids := make([]string, 0, len(destinations))
for _, destination := range destinations {
ids = append(ids, destination.ID)
}
return ids
}
func destinationSummary(destinations []config.Destination) string {
if len(destinations) == 0 {
return "none"
@@ -224,25 +381,47 @@ func destinationSummary(destinations []config.Destination) string {
}
func writeSecretConflictWarnings(w io.Writer, conflicts []config.SecretConflict) error {
return writeWarnings(w, secretConflictWarnings(conflicts))
}
func secretConflictWarnings(conflicts []config.SecretConflict) []OutputWarning {
warnings := make([]OutputWarning, 0, len(conflicts))
for _, conflict := range conflicts {
if _, err := fmt.Fprintf(w, "Warning: secret %s ignored because the real environment already has that variable\n", conflict.Name); err != nil {
return err
}
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("secret %s ignored because the real environment already has that variable", conflict.Name),
})
}
return nil
return warnings
}
func writeSSHWarnings(w io.Writer, pipeline config.Pipeline) error {
return writeWarnings(w, sshWarnings(pipeline))
}
func sshWarnings(pipeline config.Pipeline) []OutputWarning {
var warnings []OutputWarning
if pipeline.Source.Backend == config.BackendSSH && pipeline.Source.SSH.HostKeyPolicy == config.HostKeyPolicyOff {
if _, err := fmt.Fprintf(w, "Warning: pipeline=%s source host_key_policy=off disables SSH host key checking\n", pipeline.ID); err != nil {
return err
}
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("pipeline=%s source host_key_policy=off disables SSH host key checking", pipeline.ID),
})
}
for _, destination := range pipeline.Destinations {
if destination.Backend == config.BackendSSH && destination.SSH.HostKeyPolicy == config.HostKeyPolicyOff {
if _, err := fmt.Fprintf(w, "Warning: pipeline=%s destination=%s host_key_policy=off disables SSH host key checking\n", pipeline.ID, destination.ID); err != nil {
return err
}
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("pipeline=%s destination=%s host_key_policy=off disables SSH host key checking", pipeline.ID, destination.ID),
})
}
}
return warnings
}
func writeWarnings(w io.Writer, warnings []OutputWarning) error {
if w == nil {
return nil
}
for _, warning := range warnings {
if _, err := fmt.Fprintf(w, "Warning: %s\n", warning.Message); err != nil {
return err
}
}
return nil
@@ -274,6 +453,108 @@ func notifyEvent(plan publish.Plan) notify.Event {
}
}
type runResult struct {
DryRun bool `json:"dry_run"`
Pipelines []runPipelineResult `json:"pipelines"`
Actions []runActionResult `json:"actions"`
Summary runSummaryResult `json:"summary"`
}
type runPipelineResult struct {
ID string `json:"id"`
SourceBackend string `json:"source_backend"`
BundleCount int `json:"bundle_count"`
Destinations []string `json:"destinations"`
}
type runActionResult struct {
PipelineID string `json:"pipeline_id,omitempty"`
DestinationID string `json:"destination_id"`
Backend string `json:"backend"`
BundleID string `json:"bundle_id,omitempty"`
BundlePath string `json:"bundle_path"`
DestinationPath string `json:"destination_path"`
PathMapping string `json:"path_mapping,omitempty"`
Action string `json:"action"`
PrimaryURL string `json:"primary_url,omitempty"`
Reason string `json:"reason,omitempty"`
Outputs []runOutputResult `json:"outputs"`
}
type runOutputResult struct {
Path string `json:"path"`
Kind string `json:"kind"`
SourcePath string `json:"source_path,omitempty"`
Transform string `json:"transform,omitempty"`
URL string `json:"url,omitempty"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActionResult {
if planErr != nil {
destinationID := plan.DestinationID
if destinationID == "" {
destinationID = "unknown"
}
return runActionResult{
PipelineID: plan.PipelineID,
DestinationID: destinationID,
Backend: backend,
BundleID: plan.BundleID,
BundlePath: storage.DisplayPath(plan.BundlePath),
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: "error",
PrimaryURL: plan.PrimaryURL,
Reason: planErr.Error(),
Outputs: []runOutputResult{},
}
}
return runActionResult{
PipelineID: plan.PipelineID,
DestinationID: plan.DestinationID,
Backend: backend,
BundleID: plan.BundleID,
BundlePath: storage.DisplayPath(plan.BundlePath),
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: string(plan.Action),
PrimaryURL: plan.PrimaryURL,
Reason: plan.Reason,
Outputs: runOutputsFromPlan(plan.Outputs),
}
}
func errorAction(pipelineID, destinationID, backend, bundlePath string, err error) runActionResult {
return runActionResult{
PipelineID: pipelineID,
DestinationID: destinationID,
Backend: backend,
BundlePath: storage.DisplayPath(bundlePath),
DestinationPath: storage.DisplayPath(bundlePath),
Action: "error",
Reason: err.Error(),
Outputs: []runOutputResult{},
}
}
func runOutputsFromPlan(outputs []publish.Output) []runOutputResult {
results := make([]runOutputResult, 0, len(outputs))
for _, output := range outputs {
results = append(results, runOutputResult{
Path: output.DestinationPath,
Kind: output.Kind,
SourcePath: output.SourcePath,
Transform: output.Transform,
URL: output.URL,
SHA256: output.SHA256,
Size: output.Size,
})
}
return results
}
type runSummary struct {
dryRun bool
planned int
@@ -282,6 +563,7 @@ type runSummary struct {
forceReplace int
skipped int
failures int
fixedPath int
}
func (s *runSummary) recordPlan(action publish.Action) {
@@ -302,12 +584,46 @@ func (s *runSummary) recordFailure() {
s.failures++
}
func (s *runSummary) recordFixedPath() {
s.fixedPath++
}
func (s runSummary) Line() string {
status := "ok"
if s.failures > 0 {
status = "failed"
}
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d replace_older=%d force_replace=%d skipped=%d failed=%d dry_run=%t", status, s.planned, s.publishNew, s.replaceOlder, s.forceReplace, s.skipped, s.failures, s.dryRun)
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d replace_older=%d force_replace=%d skipped=%d failed=%d dry_run=%t fixed_path=%d", status, s.planned, s.publishNew, s.replaceOlder, s.forceReplace, s.skipped, s.failures, s.dryRun, s.fixedPath)
}
type runSummaryResult struct {
Status string `json:"status"`
Planned int `json:"planned"`
PublishNew int `json:"publish_new"`
ReplaceOlder int `json:"replace_older"`
ForceReplace int `json:"force_replace"`
Skipped int `json:"skipped"`
Failed int `json:"failed"`
DryRun bool `json:"dry_run"`
FixedPath int `json:"fixed_path"`
}
func (s runSummary) Result() runSummaryResult {
status := "ok"
if s.failures > 0 {
status = "failed"
}
return runSummaryResult{
Status: status,
Planned: s.planned,
PublishNew: s.publishNew,
ReplaceOlder: s.replaceOlder,
ForceReplace: s.forceReplace,
Skipped: s.skipped,
Failed: s.failures,
DryRun: s.dryRun,
FixedPath: s.fixedPath,
}
}
type runFailure struct {
@@ -343,6 +659,28 @@ func (f runFailures) Error() string {
return "run failed: " + strings.Join(parts, "; ")
}
func (f runFailures) outputErrors() []OutputError {
if len(f.items) == 0 {
return nil
}
errors := make([]OutputError, 0, len(f.items))
for _, item := range f.items {
errors = append(errors, OutputError{
PipelineID: item.pipelineID,
DestinationID: item.destinationID,
Backend: item.backend,
BundlePath: item.bundlePath,
Message: item.err.Error(),
})
}
return errors
}
func IsPartialResultError(err error) bool {
var failures runFailures
return errors.As(err, &failures)
}
func (f runFailures) Unwrap() error {
errs := make([]error, 0, len(f.items))
for _, item := range f.items {

View File

@@ -215,6 +215,380 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
if got, want := len(destinationState.Outputs), 2; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
}
if destinationState.Links != nil || destinationState.Outputs[0].URL != "" {
t.Fatalf("state links = %#v output URL=%q, want absent", destinationState.Links, destinationState.Outputs[0].URL)
}
}
func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "daily/report", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "daily", "report", "report.md"), "# Report\nSunny.\n")
if _, err := os.Stat(filepath.Join(destinationRoot, "report.md")); !os.IsNotExist(err) {
t.Fatalf("root report.md stat error = %v, want not exist", err)
}
}
func TestRunRecordsLinksForNestedBundlePath(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "daily/brentwood", testBundleOptions{})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative, "https://reports.example.com/archive", config.LinkPrimaryAuto, true, false, ""),
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, "daily", "brentwood", storage.StateFileName))
if destinationState.Links == nil || destinationState.Links.PrimaryURL != "https://reports.example.com/archive/daily/brentwood/report.md" {
t.Fatalf("state links = %#v, want source primary URL", destinationState.Links)
}
outputs := outputsByPath(destinationState.Outputs)
if outputs["report.md"].URL != "https://reports.example.com/archive/daily/brentwood/report.md" {
t.Fatalf("report URL = %q", outputs["report.md"].URL)
}
if outputs["summary.txt"].URL != "https://reports.example.com/archive/daily/brentwood/summary.txt" {
t.Fatalf("summary URL = %q", outputs["summary.txt"].URL)
}
}
func TestRunRecordsLinksForFixedIndexDestination(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "older", testBundleOptions{ID: "reports.older", Created: testutil.DefaultCreated})
writeSourceBundle(t, sourceRoot, "newer", testBundleOptions{ID: "reports.newer", Created: testutil.DefaultCreated.Add(time.Hour)})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingFixed, "https://reports.example.com/latest", config.LinkPrimaryAuto, false, true, config.TransformModeIndex),
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if destinationState.Links == nil || destinationState.Links.PrimaryURL != "https://reports.example.com/latest/" {
t.Fatalf("state links = %#v, want fixed index primary URL", destinationState.Links)
}
if got, want := len(destinationState.Outputs), 1; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
}
if destinationState.Outputs[0].Path != "index.html" || destinationState.Outputs[0].URL != "https://reports.example.com/latest/" {
t.Fatalf("state output = %#v, want index URL", destinationState.Outputs[0])
}
}
func TestRunFixedPathPublishesNewestBundleAtDestinationRoot(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{
ID: "reports.old",
Created: testutil.DefaultCreated,
Files: []testFile{
{Path: "report.md", Data: "# Report\nOld.\n"},
{Path: "summary.txt", Data: "Old summary\n"},
},
})
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
ID: "reports.new",
Created: testutil.DefaultCreated.Add(time.Hour),
Files: []testFile{
{Path: "report.md", Data: "# Report\nNew.\n"},
{Path: "summary.txt", Data: "New summary\n"},
},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
if _, err := os.Stat(filepath.Join(destinationRoot, "new", "report.md")); !os.IsNotExist(err) {
t.Fatalf("nested new report stat error = %v, want not exist", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if destinationState.Source.Manifest.ID != "reports.new" {
t.Fatalf("state source id = %q, want reports.new", destinationState.Source.Manifest.ID)
}
}
func TestRunFixedPathTieBreaksByBundlePath(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "b", testBundleOptions{
ID: "reports.b",
Created: testutil.DefaultCreated,
Files: []testFile{
{Path: "report.md", Data: "# Report\nB.\n"},
{Path: "summary.txt", Data: "B summary\n"},
},
})
writeSourceBundle(t, sourceRoot, "a", testBundleOptions{
ID: "reports.a",
Created: testutil.DefaultCreated,
Files: []testFile{
{Path: "report.md", Data: "# Report\nA.\n"},
{Path: "summary.txt", Data: "A summary\n"},
},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if destinationState.Source.Manifest.ID != "reports.a" {
t.Fatalf("state source id = %q, want reports.a", destinationState.Source.Manifest.ID)
}
}
func TestRunFixedPathDryRunReportsSelection(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{ID: "reports.old", Created: testutil.DefaultCreated})
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{ID: "reports.new", Created: testutil.DefaultCreated.Add(time.Hour)})
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
DryRun: true,
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
output := stdout.String()
for _, want := range []string{
"Warning: pipeline=reports destination=archive path_mapping=fixed candidates=2 selected_bundle=new destination_bundle=.",
"bundle=new destination=archive backend=local path_mapping=fixed target=. action=publish_new",
"fixed_path=1",
} {
if !strings.Contains(output, want) {
t.Fatalf("stdout = %q, want substring %q", output, want)
}
}
if strings.Contains(output, "bundle=old destination=archive") {
t.Fatalf("stdout = %q, older fixed candidate was planned", output)
}
if entries, err := os.ReadDir(destinationRoot); err != nil || len(entries) != 0 {
t.Fatalf("destination entries = %v err=%v, want empty", entries, err)
}
}
func TestRunFixedPathDryRunWarnsForReplacement(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{
ID: "reports.old",
Created: testutil.DefaultCreated,
Files: []testFile{
{Path: "report.md", Data: "# Report\nOld.\n"},
{Path: "summary.txt", Data: "Old summary\n"},
},
})
configPath := writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("first Run() error = %v", err)
}
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
ID: "reports.new",
Created: testutil.DefaultCreated.Add(time.Hour),
Files: []testFile{
{Path: "report.md", Data: "# Report\nNew.\n"},
{Path: "summary.txt", Data: "New summary\n"},
},
})
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: configPath,
DryRun: true,
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
output := stdout.String()
for _, want := range []string{
"Warning: pipeline=reports destination=archive path_mapping=fixed action=replace_older replaces destination root for selected_bundle=new",
"bundle=new destination=archive backend=local path_mapping=fixed target=. action=replace_older",
} {
if !strings.Contains(output, want) {
t.Fatalf("stdout = %q, want substring %q", output, want)
}
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
}
func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{
ID: "reports.old",
Created: testutil.DefaultCreated,
Files: []testFile{
{Path: "report.md", Data: "# Report\nOld.\n"},
{Path: "summary.txt", Data: "Old summary\n"},
},
})
configPath := writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("first Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
ID: "reports.new",
Created: testutil.DefaultCreated.Add(time.Hour),
Files: []testFile{
{Path: "report.md", Data: "# Report\nNew.\n"},
{Path: "summary.txt", Data: "New summary\n"},
},
})
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("second Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if destinationState.Source.Manifest.ID != "reports.new" {
t.Fatalf("state source id = %q, want reports.new", destinationState.Source.Manifest.ID)
}
}
func TestRunFixedPathSkipsWhenDestinationStateIsNewer(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
newer := testutil.ValidManifest(testutil.BundleOptions{
ID: "reports.newer",
Created: testutil.DefaultCreated.Add(time.Hour),
})
writeDestinationState(t, destinationRoot, "", newer)
if err := os.WriteFile(filepath.Join(destinationRoot, "report.md"), []byte("# Report\nExisting.\n"), 0o600); err != nil {
t.Fatalf("write existing report: %v", err)
}
writeSourceBundle(t, sourceRoot, "older", testBundleOptions{
ID: "reports.older",
Created: testutil.DefaultCreated,
})
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(stdout.String(), "action=skip_destination_newer") {
t.Fatalf("stdout = %q, want skip_destination_newer", stdout.String())
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nExisting.\n")
}
func TestRunFixedPathFailsUnmanagedWithoutForce(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "bundle", testBundleOptions{})
if err := os.WriteFile(filepath.Join(destinationRoot, "unmanaged.txt"), []byte("data"), 0o600); err != nil {
t.Fatalf("write unmanaged file: %v", err)
}
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
if err == nil || !strings.Contains(err.Error(), "fail_unmanaged") {
t.Fatalf("Run() error = %v, want unmanaged failure", err)
}
}
func TestRunFixedPathForceReplacementStaysWithinDestinationRoot(t *testing.T) {
sourceRoot := t.TempDir()
parent := t.TempDir()
destinationRoot := filepath.Join(parent, "latest")
if err := os.MkdirAll(destinationRoot, 0o755); err != nil {
t.Fatalf("mkdir destination: %v", err)
}
if err := os.WriteFile(filepath.Join(parent, "keep.txt"), []byte("keep"), 0o600); err != nil {
t.Fatalf("write sibling: %v", err)
}
if err := os.WriteFile(filepath.Join(destinationRoot, "unmanaged.txt"), []byte("old"), 0o600); err != nil {
t.Fatalf("write unmanaged: %v", err)
}
writeSourceBundle(t, sourceRoot, "bundle", testBundleOptions{})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
Force: true,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
assertFile(t, filepath.Join(parent, "keep.txt"), "keep")
if _, err := os.Stat(filepath.Join(destinationRoot, "unmanaged.txt")); !os.IsNotExist(err) {
t.Fatalf("unmanaged stat error = %v, want removed", err)
}
}
func TestRunFixedPathRemoteBackendsUseBackendRoots(t *testing.T) {
localSourceRoot := t.TempDir()
writeSourceBundle(t, localSourceRoot, "old", testBundleOptions{
ID: "reports.old",
Created: testutil.DefaultCreated,
Files: []testFile{
{Path: "report.md", Data: "# Report\nOld.\n"},
{Path: "summary.txt", Data: "Old summary\n"},
},
})
writeSourceBundle(t, localSourceRoot, "new", testBundleOptions{
ID: "reports.new",
Created: testutil.DefaultCreated.Add(time.Hour),
Files: []testFile{
{Path: "report.md", Data: "# Report\nNew.\n"},
{Path: "summary.txt", Data: "New summary\n"},
},
})
s3Destination := fake.New()
sshDestination := fake.New()
cfg := config.Config{Pipelines: []config.Pipeline{{
ID: "reports",
Source: config.Backend{Backend: config.BackendLocal, Path: localSourceRoot},
Destinations: []config.Destination{
{
ID: "object-latest",
Backend: config.BackendS3,
Endpoint: "http://s3.test",
Bucket: "destination-bucket",
PathMap: config.PathMapping{Mode: config.PathMappingFixed},
},
{
ID: "ssh-latest",
Backend: config.BackendSSH,
Host: "ssh.test",
Path: "/latest",
PathMap: config.PathMapping{Mode: config.PathMappingFixed},
},
},
}}}
config.ApplyDefaults(&cfg)
provider := fakeBackendFactoryProvider(t, map[string]storage.Backend{
"s3:destination-bucket": s3Destination,
"ssh:/latest": sshDestination,
})
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{}, provider); err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFakeFile(t, s3Destination, "report.md", "# Report\nNew.\n")
assertFakeFile(t, s3Destination, "summary.txt", "New summary\n")
assertFakeMissing(t, s3Destination, "new/report.md")
assertFakeFile(t, sshDestination, "report.md", "# Report\nNew.\n")
assertFakeFile(t, sshDestination, "summary.txt", "New summary\n")
assertFakeMissing(t, sshDestination, "new/report.md")
}
func TestRunNotifiesAfterPublication(t *testing.T) {
@@ -372,6 +746,59 @@ func TestRunPublishesHTMLOnly(t *testing.T) {
}
}
func TestRunPublishesHTMLIndexWithExplicitInput(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{
ExtraFiles: []testFile{{Path: "notes.md", Data: "# Notes\nHidden.\n"}},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "report.md")})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<h1>Report</h1>")
if _, err := os.Stat(filepath.Join(destinationRoot, "report.html")); !os.IsNotExist(err) {
t.Fatalf("report.html stat error = %v, want not exist", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if got, want := len(destinationState.Outputs), 1; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
}
output := destinationState.Outputs[0]
if output.Kind != state.OutputKindGenerated || output.Transform != "markdown_to_html" || output.Path != "index.html" || output.SourcePath != "report.md" {
t.Fatalf("generated output metadata = %#v", output)
}
}
func TestRunPublishesHTMLIndexWithSingleMarkdownFallback(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<h1>Report</h1>")
}
func TestRunFailsIndexModeWithAmbiguousMarkdownInput(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{
ExtraFiles: []testFile{{Path: "notes.md", Data: "# Notes\n"}},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")})
if err == nil || !strings.Contains(err.Error(), "multiple markdown source files") {
t.Fatalf("Run() error = %v, want ambiguous input error", err)
}
if entries, err := os.ReadDir(destinationRoot); err != nil || len(entries) != 0 {
t.Fatalf("destination entries = %v err=%v, want empty", entries, err)
}
}
func TestRunPublishesSourceAndHTML(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
@@ -427,6 +854,22 @@ func TestRunFailsOnOutputPathCollision(t *testing.T) {
}
}
func TestRunFailsOnIndexOutputPathCollision(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{
ExtraFiles: []testFile{{Path: "index.html", Data: "<p>source index</p>\n"}},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, true, true, config.TransformModeIndex, "report.md")})
if err == nil || !strings.Contains(err.Error(), "destination output path collision") {
t.Fatalf("Run() error = %v, want collision", err)
}
if entries, err := os.ReadDir(destinationRoot); err != nil || len(entries) != 0 {
t.Fatalf("destination entries = %v err=%v, want empty", entries, err)
}
}
func TestRunDryRunReportsGeneratedOutputs(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
@@ -446,6 +889,73 @@ func TestRunDryRunReportsGeneratedOutputs(t *testing.T) {
}
}
func TestRunDryRunReportsIndexOutputWithoutWriting(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, ""),
DryRun: true,
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(stdout.String(), "outputs=index.html") {
t.Fatalf("stdout = %q, want index output path", stdout.String())
}
if entries, err := os.ReadDir(destinationRoot); err != nil || len(entries) != 0 {
t.Fatalf("destination entries = %v err=%v, want empty", entries, err)
}
}
func TestRunSourceOnlyDoesNotWriteIndexOutput(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, true, false, config.TransformModeIndex, "")})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
if _, err := os.Stat(filepath.Join(destinationRoot, "index.html")); !os.IsNotExist(err) {
t.Fatalf("index.html stat error = %v, want not exist", err)
}
}
func TestRunReplacesHTMLIndexOutput(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{
Created: testutil.DefaultCreated,
Files: []testFile{
{Path: "report.md", Data: "# Report\nOld.\n"},
{Path: "summary.txt", Data: "Summary\n"},
},
})
configPath := writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("first Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<p>Old.</p>")
writeSourceBundle(t, sourceRoot, "", testBundleOptions{
Created: testutil.DefaultCreated.Add(time.Hour),
Files: []testFile{
{Path: "report.md", Data: "# Report\nNew.\n"},
{Path: "summary.txt", Data: "Summary\n"},
},
})
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("second Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<p>New.</p>")
}
func TestRunSkipsWhenDestinationStateMatches(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
@@ -727,6 +1237,7 @@ func TestRunDryRunDoesNotWrite(t *testing.T) {
type testBundleOptions struct {
ID string
Created time.Time
Files []testFile
ExtraFiles []testFile
}
@@ -737,6 +1248,13 @@ type testFile struct {
func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptions) bundle.Manifest {
t.Helper()
var files []testutil.SourceFile
if opts.Files != nil {
files = make([]testutil.SourceFile, 0, len(opts.Files))
for _, file := range opts.Files {
files = append(files, testutil.SourceFile{Path: file.Path, Data: file.Data})
}
}
extraFiles := make([]testutil.SourceFile, 0, len(opts.ExtraFiles))
for _, file := range opts.ExtraFiles {
extraFiles = append(extraFiles, testutil.SourceFile{Path: file.Path, Data: file.Data})
@@ -744,6 +1262,7 @@ func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptio
return testutil.WriteSourceBundle(t, root, relative, testutil.BundleOptions{
ID: opts.ID,
Created: opts.Created,
Files: files,
ExtraFiles: extraFiles,
})
}
@@ -779,6 +1298,82 @@ pipelines:
`)
}
func writeLocalConfigWithPathMapping(t *testing.T, sourceRoot, destinationRoot, mode string) string {
t.Helper()
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
path_mapping:
mode: `+mode+`
`)
}
func writeLocalConfigWithLinks(t *testing.T, sourceRoot, destinationRoot, pathMapping, baseURL, primary string, publishSource, publishHTML bool, transformMode string) string {
t.Helper()
transformConfig := ""
if publishHTML {
transformConfig = `
transform:
markdown_to_html:
enabled: true
mode: ` + transformMode
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
path_mapping:
mode: `+pathMapping+`
links:
base_url: `+baseURL+`
primary: `+primary+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
`)
}
func writeLocalConfigWithMarkdownTransform(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool, mode, input string) string {
t.Helper()
enabled := publishHTML
inputConfig := ""
if input != "" {
inputConfig = `
input: ` + input
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+`
transform:
markdown_to_html:
enabled: `+fmt.Sprintf("%t", enabled)+`
mode: `+mode+inputConfig+`
`)
}
func writeFanoutConfig(t *testing.T, sourceRoot, firstDestination, secondDestination string) string {
t.Helper()
return testutil.WriteFanoutLocalConfig(t, sourceRoot, firstDestination, secondDestination)
@@ -831,6 +1426,14 @@ func readStateFile(t *testing.T, path string) state.DistributorState {
return testutil.ReadDestinationState(t, path)
}
func outputsByPath(outputs []state.OutputFile) map[string]state.OutputFile {
byPath := make(map[string]state.OutputFile, len(outputs))
for _, output := range outputs {
byPath[output.Path] = output
}
return byPath
}
func assertFile(t *testing.T, path, want string) {
t.Helper()
data, err := os.ReadFile(path)

View File

@@ -0,0 +1,120 @@
package app
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
type sourceCommandOptions struct {
CommandName string
Path string
ConfigPath string
PipelineID string
BundlePath string
}
type sourceSelection struct {
Bundles []bundle.Bundle
PipelineID string
SourceBackend string
ConfigMode bool
Warnings []OutputWarning
}
func selectSourceBundles(ctx context.Context, options sourceCommandOptions, provider backendFactoryProvider) (sourceSelection, error) {
if err := ctx.Err(); err != nil {
return sourceSelection{}, err
}
if options.ConfigPath != "" {
cfg, err := config.LoadFile(options.ConfigPath)
if err != nil {
return sourceSelection{}, err
}
return selectSourceBundlesFromConfig(ctx, cfg, options, provider)
}
if options.PipelineID != "" {
return sourceSelection{}, fmt.Errorf("configured source mode requires --config")
}
if options.BundlePath != "" {
return sourceSelection{}, fmt.Errorf("configured source mode requires --config")
}
if options.Path == "" {
return sourceSelection{}, fmt.Errorf("%s command requires a path", options.CommandName)
}
backend, err := newBackendFactory().openLocalPath(ctx, options.Path)
if err != nil {
return sourceSelection{}, err
}
defer closeBackend(backend)
bundles, err := bundle.Discover(ctx, backend, "")
if err != nil {
return sourceSelection{}, err
}
return sourceSelection{Bundles: bundles}, nil
}
func selectSourceBundlesFromConfig(ctx context.Context, cfg config.Config, options sourceCommandOptions, provider backendFactoryProvider) (sourceSelection, error) {
if options.Path != "" {
return sourceSelection{}, fmt.Errorf("configured source mode does not accept a local path")
}
if options.PipelineID == "" {
return sourceSelection{}, fmt.Errorf("configured source mode requires --pipeline")
}
secretLoad, err := config.LoadSecretEnvironment(cfg.Secrets.Directory, nil)
if err != nil {
return sourceSelection{}, err
}
pipeline, ok := findPipeline(cfg, options.PipelineID)
if !ok {
return sourceSelection{}, fmt.Errorf("pipeline %q not found", options.PipelineID)
}
backends := provider(secretLoad.Environment)
sourceBackend, err := backends.openSource(ctx, pipeline.Source)
if err != nil {
return sourceSelection{}, fmt.Errorf("pipeline %s source backend %s: %w", pipeline.ID, pipeline.Source.Backend, err)
}
defer closeBackend(sourceBackend)
var bundles []bundle.Bundle
if options.BundlePath != "" {
sourceBundle, err := bundle.Validate(ctx, sourceBackend, options.BundlePath)
if err != nil {
return sourceSelection{}, fmt.Errorf("pipeline %s source backend %s bundle %s: %w", pipeline.ID, pipeline.Source.Backend, storage.DisplayPath(options.BundlePath), err)
}
bundles = []bundle.Bundle{sourceBundle}
} else {
bundles, err = bundle.Discover(ctx, sourceBackend, "")
if err != nil {
return sourceSelection{}, fmt.Errorf("pipeline %s source backend %s discover source bundles: %w", pipeline.ID, pipeline.Source.Backend, err)
}
}
return sourceSelection{
Bundles: bundles,
PipelineID: pipeline.ID,
SourceBackend: pipeline.Source.Backend,
ConfigMode: true,
Warnings: append(secretConflictWarnings(secretLoad.Conflicts), sourceSSHWarnings(pipeline)...),
}, nil
}
func findPipeline(cfg config.Config, id string) (config.Pipeline, bool) {
for _, pipeline := range cfg.Pipelines {
if pipeline.ID == id {
return pipeline, true
}
}
return config.Pipeline{}, false
}
func sourceSSHWarnings(pipeline config.Pipeline) []OutputWarning {
if pipeline.Source.Backend != config.BackendSSH || pipeline.Source.SSH.HostKeyPolicy != config.HostKeyPolicyOff {
return nil
}
return []OutputWarning{{
Message: fmt.Sprintf("pipeline=%s source host_key_policy=off disables SSH host key checking", pipeline.ID),
}}
}

View File

@@ -5,28 +5,97 @@ import (
"fmt"
"io"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
type ValidateOptions struct {
Path string
Stdout io.Writer
Path string
ConfigPath string
PipelineID string
BundlePath string
Stdout io.Writer
OutputFormat OutputFormat
}
func Validate(ctx context.Context, options ValidateOptions) error {
if options.Path == "" {
return fmt.Errorf("validate command requires a path")
return validateWithBackendFactory(ctx, options, newBackendFactoryWithEnvironment)
}
func validateWithBackendFactory(ctx context.Context, options ValidateOptions, provider backendFactoryProvider) error {
if err := ValidateOutputFormat(options.OutputFormat); err != nil {
return err
}
backend, err := newBackendFactory().openLocalPath(ctx, options.Path)
selection, err := selectSourceBundles(ctx, sourceCommandOptions{
CommandName: "validate",
Path: options.Path,
ConfigPath: options.ConfigPath,
PipelineID: options.PipelineID,
BundlePath: options.BundlePath,
}, provider)
if err != nil {
return err
}
bundles, err := bundle.Discover(ctx, backend, "")
return writeValidateResult(options, selection)
}
func validateConfigWithBackendFactory(ctx context.Context, cfg config.Config, options ValidateOptions, provider backendFactoryProvider) error {
if err := ValidateOutputFormat(options.OutputFormat); err != nil {
return err
}
selection, err := selectSourceBundlesFromConfig(ctx, cfg, sourceCommandOptions{
CommandName: "validate",
PipelineID: options.PipelineID,
BundlePath: options.BundlePath,
}, provider)
if err != nil {
return err
}
return writeValidateResult(options, selection)
}
func writeValidateResult(options ValidateOptions, selection sourceSelection) error {
if IsJSONOutput(options.OutputFormat) {
return WriteJSONEnvelope(options.Stdout, "validate", true, selection.Warnings, validateResultFromSelection(selection), nil)
}
var err error
if options.Stdout != nil {
_, err = fmt.Fprintf(options.Stdout, "Validated %d bundle(s)\n", len(bundles))
if err := writeWarnings(options.Stdout, selection.Warnings); err != nil {
return err
}
if selection.ConfigMode {
_, err = fmt.Fprintf(options.Stdout, "Validated %d bundle(s) for pipeline %s source %s\n", len(selection.Bundles), selection.PipelineID, selection.SourceBackend)
} else {
_, err = fmt.Fprintf(options.Stdout, "Validated %d bundle(s)\n", len(selection.Bundles))
}
}
return err
}
type validateResult struct {
PipelineID string `json:"pipeline_id,omitempty"`
SourceBackend string `json:"source_backend,omitempty"`
BundleCount int `json:"bundle_count"`
Bundles []validateBundleResult `json:"bundles"`
}
type validateBundleResult struct {
Path string `json:"path"`
ID string `json:"id"`
}
func validateResultFromSelection(selection sourceSelection) validateResult {
result := validateResult{
PipelineID: selection.PipelineID,
SourceBackend: selection.SourceBackend,
BundleCount: len(selection.Bundles),
Bundles: make([]validateBundleResult, 0, len(selection.Bundles)),
}
for _, sourceBundle := range selection.Bundles {
result.Bundles = append(result.Bundles, validateBundleResult{
Path: storage.DisplayPath(sourceBundle.RootRelativePath),
ID: sourceBundle.Manifest.ID,
})
}
return result
}

View File

@@ -3,9 +3,15 @@ package app
import (
"bytes"
"context"
"encoding/json"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
)
func TestValidateLocalBundle(t *testing.T) {
@@ -31,9 +37,182 @@ func TestValidateExampleSourceBundle(t *testing.T) {
}
}
func TestValidateConfiguredLocalSource(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
var stdout bytes.Buffer
err := Validate(context.Background(), ValidateOptions{
ConfigPath: testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot),
PipelineID: "reports",
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Validate() configured source error = %v", err)
}
if got, want := stdout.String(), "Validated 1 bundle(s) for pipeline reports source local\n"; got != want {
t.Fatalf("stdout = %q, want %q", got, want)
}
}
func TestValidateConfiguredSourceBundlePath(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "daily/one", testutil.BundleOptions{ID: "reports.one"})
testutil.WriteSourceBundle(t, sourceRoot, "daily/two", testutil.BundleOptions{ID: "reports.two"})
var stdout bytes.Buffer
err := Validate(context.Background(), ValidateOptions{
ConfigPath: testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot),
PipelineID: "reports",
BundlePath: "daily/two",
Stdout: &stdout,
OutputFormat: OutputFormatJSON,
})
if err != nil {
t.Fatalf("Validate() configured bundle error = %v", err)
}
result := decodeAppResult(t, stdout.String())
if result["pipeline_id"] != "reports" || result["source_backend"] != "local" || result["bundle_count"] != float64(1) {
t.Fatalf("result = %#v, want configured source summary", result)
}
bundles, ok := result["bundles"].([]any)
if !ok || len(bundles) != 1 {
t.Fatalf("bundles = %#v, want one bundle", result["bundles"])
}
sourceBundle, ok := bundles[0].(map[string]any)
if !ok || sourceBundle["path"] != "daily/two" || sourceBundle["id"] != "reports.two" {
t.Fatalf("bundle = %#v, want narrowed bundle", sourceBundle)
}
}
func TestValidateConfiguredRemoteSourcesThroughStorageAbstraction(t *testing.T) {
s3Source := fake.New()
testutil.WriteFakeSourceBundle(t, s3Source, "", testutil.BundleOptions{ID: "reports.s3"})
sshSource := fake.New()
testutil.WriteFakeSourceBundle(t, sshSource, "daily", testutil.BundleOptions{ID: "reports.ssh"})
cfg := config.Config{Pipelines: []config.Pipeline{
{
ID: "s3-reports",
Source: config.Backend{
Backend: config.BackendS3,
Endpoint: "http://s3.test",
Bucket: "source-bucket",
},
Destinations: []config.Destination{{
ID: "archive",
Backend: config.BackendLocal,
Path: t.TempDir(),
}},
},
{
ID: "ssh-reports",
Source: config.Backend{
Backend: config.BackendSSH,
Host: "ssh.test",
Path: "/source",
},
Destinations: []config.Destination{{
ID: "archive",
Backend: config.BackendLocal,
Path: t.TempDir(),
}},
},
}}
config.ApplyDefaults(&cfg)
provider := fakeBackendFactoryProvider(t, map[string]storage.Backend{
"s3:source-bucket": s3Source,
"ssh:/source": sshSource,
})
var s3Stdout bytes.Buffer
if err := validateConfigWithBackendFactory(context.Background(), cfg, ValidateOptions{
PipelineID: "s3-reports",
Stdout: &s3Stdout,
OutputFormat: OutputFormatJSON,
}, provider); err != nil {
t.Fatalf("validate s3 source error = %v", err)
}
s3Result := decodeAppResult(t, s3Stdout.String())
if s3Result["source_backend"] != "s3" || s3Result["bundle_count"] != float64(1) {
t.Fatalf("s3 result = %#v, want one s3 bundle", s3Result)
}
var sshStdout bytes.Buffer
if err := validateConfigWithBackendFactory(context.Background(), cfg, ValidateOptions{
PipelineID: "ssh-reports",
BundlePath: "daily",
Stdout: &sshStdout,
}, provider); err != nil {
t.Fatalf("validate ssh source error = %v", err)
}
if !strings.Contains(sshStdout.String(), "pipeline ssh-reports source ssh") {
t.Fatalf("ssh stdout = %q, want ssh source summary", sshStdout.String())
}
}
func TestValidateConfiguredSourceLoadsSecretsBeforeOpeningBackend(t *testing.T) {
sourceRoot := filepath.Join(t.TempDir(), "missing-source")
destinationRoot := t.TempDir()
configPath := writeConfigFile(t, `
secrets:
directory: `+filepath.Join(t.TempDir(), "missing-secrets")+`
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
`)
err := Validate(context.Background(), ValidateOptions{ConfigPath: configPath, PipelineID: "reports"})
if err == nil {
t.Fatal("Validate() error = nil, want secrets directory error")
}
if !strings.Contains(err.Error(), "load secrets directory") {
t.Fatalf("Validate() error = %v, want secrets directory error", err)
}
if strings.Contains(err.Error(), "missing-source") {
t.Fatalf("Validate() error = %v, opened source before loading secrets", err)
}
}
func TestValidateConfiguredSourceRequiresPipeline(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
err := Validate(context.Background(), ValidateOptions{
ConfigPath: testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot),
})
if err == nil || !strings.Contains(err.Error(), "requires --pipeline") {
t.Fatalf("Validate() error = %v, want required pipeline", err)
}
}
func TestValidateRequiresPath(t *testing.T) {
err := Validate(context.Background(), ValidateOptions{})
if err == nil || !strings.Contains(err.Error(), "requires a path") {
t.Fatalf("Validate() error = %v, want required path", err)
}
}
func decodeAppResult(t *testing.T, output string) map[string]any {
t.Helper()
var envelope map[string]any
if err := json.Unmarshal([]byte(output), &envelope); err != nil {
t.Fatalf("decode output: %v; output = %q", err, output)
}
result, ok := envelope["result"].(map[string]any)
if !ok {
t.Fatalf("result = %#v, want object", envelope["result"])
}
return result
}

View File

@@ -1,12 +1,10 @@
package bundle
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"regexp"
"strconv"
"strings"
publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
)
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
@@ -19,31 +17,13 @@ func ValidateDigest(value string) error {
}
func FileDigest(data []byte) string {
sum := sha256.Sum256(data)
return "sha256:" + hex.EncodeToString(sum[:])
return publicbundle.FileDigest(data)
}
func BundleDigest(files []ManifestFile) string {
canonical := CanonicalFilePayload(files)
sum := sha256.Sum256([]byte(canonical))
return "sha256:" + hex.EncodeToString(sum[:])
return publicbundle.BundleDigest(files)
}
func CanonicalFilePayload(files []ManifestFile) string {
var builder strings.Builder
builder.WriteByte('[')
for index, file := range files {
if index > 0 {
builder.WriteByte(',')
}
builder.WriteString(`{"path":`)
builder.WriteString(strconv.Quote(file.Path))
builder.WriteString(`,"sha256":`)
builder.WriteString(strconv.Quote(file.SHA256))
builder.WriteString(`,"size":`)
builder.WriteString(strconv.FormatInt(file.Size, 10))
builder.WriteByte('}')
}
builder.WriteByte(']')
return builder.String()
return publicbundle.CanonicalFilePayload(files)
}

View File

@@ -1,110 +1,32 @@
package bundle
import (
"bytes"
"encoding/json"
"fmt"
"io"
"time"
)
import publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
const ManifestName = "manifest.json"
const ManifestName = publicbundle.ManifestName
type Manifest struct {
SchemaVersion int `json:"schema_version"`
ID string `json:"id"`
Digest string `json:"digest"`
Created time.Time `json:"created"`
Files []ManifestFile `json:"files"`
}
const SchemaVersion = publicbundle.SchemaVersion
type ManifestFile struct {
Path string `json:"path"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
type Manifest = publicbundle.Manifest
type ManifestFile = publicbundle.ManifestFile
type Bundle struct {
RootRelativePath string
Manifest Manifest
}
type rawManifest struct {
SchemaVersion *int `json:"schema_version"`
ID *string `json:"id"`
Digest *string `json:"digest"`
Created *string `json:"created"`
Files []rawManifestFile `json:"files"`
}
type rawManifestFile struct {
Path *string `json:"path"`
SHA256 *string `json:"sha256"`
Size *int64 `json:"size"`
}
func ParseManifest(data []byte) (Manifest, error) {
decoder := json.NewDecoder(bytes.NewReader(data))
var raw rawManifest
if err := decoder.Decode(&raw); err != nil {
return Manifest{}, fmt.Errorf("parse manifest: %w", err)
}
var extra any
if err := decoder.Decode(&extra); err != io.EOF {
return Manifest{}, fmt.Errorf("parse manifest: trailing data")
}
var manifest Manifest
if raw.SchemaVersion == nil {
return Manifest{}, fmt.Errorf("manifest schema_version is required")
}
manifest.SchemaVersion = *raw.SchemaVersion
if raw.ID == nil || *raw.ID == "" {
return Manifest{}, fmt.Errorf("manifest id is required")
}
manifest.ID = *raw.ID
if raw.Digest == nil || *raw.Digest == "" {
return Manifest{}, fmt.Errorf("manifest digest is required")
}
manifest.Digest = *raw.Digest
if raw.Created == nil || *raw.Created == "" {
return Manifest{}, fmt.Errorf("manifest created is required")
}
created, err := time.Parse(time.RFC3339, *raw.Created)
if err != nil {
return Manifest{}, fmt.Errorf("manifest created must be RFC3339: %w", err)
}
manifest.Created = created
if len(raw.Files) == 0 {
return Manifest{}, fmt.Errorf("manifest files is required")
}
for index, rawFile := range raw.Files {
file, err := parseManifestFile(index, rawFile)
if err != nil {
return Manifest{}, err
}
manifest.Files = append(manifest.Files, file)
}
if err := ValidateManifest(manifest); err != nil {
return Manifest{}, fmt.Errorf("manifest %w", err)
}
return manifest, nil
return publicbundle.ParseManifest(data)
}
func parseManifestFile(index int, raw rawManifestFile) (ManifestFile, error) {
if raw.Path == nil || *raw.Path == "" {
return ManifestFile{}, fmt.Errorf("manifest files[%d].path is required", index)
}
if raw.SHA256 == nil || *raw.SHA256 == "" {
return ManifestFile{}, fmt.Errorf("manifest files[%d].sha256 is required", index)
}
if raw.Size == nil {
return ManifestFile{}, fmt.Errorf("manifest files[%d].size is required", index)
}
return ManifestFile{
Path: *raw.Path,
SHA256: *raw.SHA256,
Size: *raw.Size,
}, nil
func MarshalManifest(manifest Manifest) ([]byte, error) {
return publicbundle.MarshalManifest(manifest)
}
func ValidateManifest(manifest Manifest) error {
return publicbundle.ValidateManifest(manifest)
}
func ValidateSourcePath(path string) error {
return publicbundle.ValidateSourcePath(path)
}

View File

@@ -7,55 +7,6 @@ import (
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
func ValidateSourcePath(path string) error {
if err := storage.ValidatePath(path); err != nil {
return err
}
switch path {
case ManifestName, storage.StateFileName:
return fmt.Errorf("%q is reserved", path)
}
return nil
}
func ValidateManifest(manifest Manifest) error {
if manifest.SchemaVersion != 1 {
return fmt.Errorf("schema_version must be 1")
}
if manifest.ID == "" {
return fmt.Errorf("id is required")
}
if err := ValidateDigest(manifest.Digest); err != nil {
return fmt.Errorf("digest: %w", err)
}
if manifest.Created.IsZero() {
return fmt.Errorf("created is required")
}
if len(manifest.Files) == 0 {
return fmt.Errorf("files is required")
}
seen := make(map[string]struct{}, len(manifest.Files))
for index, file := range manifest.Files {
if err := ValidateSourcePath(file.Path); err != nil {
return fmt.Errorf("files[%d].path: %w", index, err)
}
if err := ValidateDigest(file.SHA256); err != nil {
return fmt.Errorf("files[%d].sha256: %w", index, err)
}
if file.Size < 0 {
return fmt.Errorf("files[%d].size must be non-negative", index)
}
if _, exists := seen[file.Path]; exists {
return fmt.Errorf("files[%d].path duplicates %q", index, file.Path)
}
seen[file.Path] = struct{}{}
}
if actual := BundleDigest(manifest.Files); actual != manifest.Digest {
return fmt.Errorf("digest mismatch: got %s want %s", actual, manifest.Digest)
}
return nil
}
func Validate(ctx context.Context, backend storage.Backend, bundleRoot string) (Bundle, error) {
return validateAt(ctx, backend, bundleRoot, bundleRoot)
}

22
internal/cli/format.go Normal file
View File

@@ -0,0 +1,22 @@
package cli
import (
"flag"
"fmt"
"io"
"gitea.maximumdirect.net/eric/distributor/internal/app"
)
func addFormatFlag(flags *flag.FlagSet) *string {
return flags.String("format", string(app.OutputFormatText), "output format: text or json")
}
func parseOutputFormat(stderr io.Writer, command, raw string) (app.OutputFormat, bool) {
format := app.OutputFormat(raw)
if err := app.ValidateOutputFormat(format); err != nil {
fmt.Fprintf(stderr, "%s: %s --format: %s\n", app.Name, command, err)
return "", false
}
return app.NormalizeOutputFormat(format), true
}

View File

@@ -2,6 +2,7 @@ package cli
import (
"context"
"flag"
"fmt"
"io"
@@ -13,11 +14,34 @@ func inspectCommand(ctx context.Context, args []string, stdout, stderr io.Writer
printInspectHelp(stdout)
return exitOK
}
path, ok := parseOptionalPathArg(stderr, "inspect", args)
flags := flag.NewFlagSet("inspect", flag.ContinueOnError)
flags.SetOutput(stderr)
configPath := flags.String("config", "", "path to config file")
pipelineID := flags.String("pipeline", "", "pipeline id")
bundlePath := flags.String("bundle", "", "source-root-relative bundle path")
formatFlag := addFormatFlag(flags)
if err := flags.Parse(args); err != nil {
return exitUsage
}
format, ok := parseOutputFormat(stderr, "inspect", *formatFlag)
if !ok {
return exitUsage
}
if err := app.Inspect(ctx, app.InspectOptions{Path: path, Stdout: stdout}); err != nil {
path, ok := parseOptionalPathArg(stderr, "inspect", flags.Args())
if !ok {
return exitUsage
}
if !validateInspectModeOK(stderr, "inspect", path, *configPath, *pipelineID, *bundlePath) {
return exitUsage
}
if err := app.Inspect(ctx, app.InspectOptions{
Path: path,
ConfigPath: *configPath,
PipelineID: *pipelineID,
BundlePath: *bundlePath,
Stdout: stdout,
OutputFormat: format,
}); err != nil {
return fail(stderr, err)
}
return exitOK
@@ -25,8 +49,16 @@ func inspectCommand(ctx context.Context, args []string, stdout, stderr io.Writer
func printInspectHelp(w io.Writer) {
fmt.Fprint(w, `Usage:
distributor inspect <path>
distributor inspect [--format text|json] <path>
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
Print a normalized summary of local source bundles.
Options:
--config <path> Path to config file for configured source inspection
--pipeline <id> Pipeline id to inspect in config mode
--bundle <path> Source-root-relative bundle path to inspect
--format text|json Output format
Print a normalized summary of local source bundles or a configured pipeline
source.
`)
}

139
internal/cli/manifest.go Normal file
View File

@@ -0,0 +1,139 @@
package cli
import (
"context"
"flag"
"fmt"
"io"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/app"
)
func manifestCommand(ctx context.Context, args []string, stdout, stderr io.Writer) int {
if len(args) == 0 || args[0] == "-h" || args[0] == "--help" || args[0] == "help" {
printManifestHelp(stdout)
return exitOK
}
switch args[0] {
case "create":
return manifestCreateCommand(ctx, args[1:], stdout, stderr)
default:
fmt.Fprintf(stderr, "%s: manifest unknown command %q\n\n", app.Name, args[0])
printManifestHelp(stderr)
return exitUsage
}
}
func manifestCreateCommand(ctx context.Context, args []string, stdout, stderr io.Writer) int {
if hasHelp(args) {
printManifestCreateHelp(stdout)
return exitOK
}
flags := flag.NewFlagSet("manifest create", flag.ContinueOnError)
flags.SetOutput(stderr)
id := flags.String("id", "", "source bundle id")
created := flags.String("created", "", "source created timestamp")
overwrite := flags.Bool("overwrite", false, "replace an existing manifest.json")
formatFlag := addFormatFlag(flags)
var files repeatedFlag
flags.Var(&files, "file", "bundle-relative file to include")
flagArgs, positionalArgs, ok := splitManifestCreateArgs(stderr, args)
if !ok {
return exitUsage
}
if err := flags.Parse(flagArgs); err != nil {
return exitUsage
}
if len(positionalArgs) != 1 {
fmt.Fprintf(stderr, "%s: manifest create requires exactly one bundle path\n", app.Name)
return exitUsage
}
format, ok := parseOutputFormat(stderr, "manifest create", *formatFlag)
if !ok {
return exitUsage
}
err := app.ManifestCreate(ctx, app.ManifestCreateOptions{
Root: positionalArgs[0],
ID: *id,
Created: *created,
Files: []string(files),
Overwrite: *overwrite,
Stdout: stdout,
OutputFormat: format,
})
if err != nil {
return fail(stderr, err)
}
return exitOK
}
func splitManifestCreateArgs(stderr io.Writer, args []string) ([]string, []string, bool) {
var flagArgs []string
var positionalArgs []string
for index := 0; index < len(args); index++ {
arg := args[index]
switch arg {
case "--overwrite":
flagArgs = append(flagArgs, arg)
case "--id", "--created", "--file", "--format":
if index+1 >= len(args) {
fmt.Fprintf(stderr, "%s: manifest create %s requires a value\n", app.Name, arg)
return nil, nil, false
}
flagArgs = append(flagArgs, arg, args[index+1])
index++
default:
if strings.HasPrefix(arg, "--id=") ||
strings.HasPrefix(arg, "--created=") ||
strings.HasPrefix(arg, "--file=") ||
strings.HasPrefix(arg, "--format=") {
flagArgs = append(flagArgs, arg)
continue
}
if strings.HasPrefix(arg, "-") {
flagArgs = append(flagArgs, arg)
continue
}
positionalArgs = append(positionalArgs, arg)
}
}
return flagArgs, positionalArgs, true
}
type repeatedFlag []string
func (f *repeatedFlag) String() string {
return fmt.Sprint([]string(*f))
}
func (f *repeatedFlag) Set(value string) error {
*f = append(*f, value)
return nil
}
func printManifestHelp(w io.Writer) {
fmt.Fprint(w, `Usage:
distributor manifest <command> [options]
Commands:
create Create a source bundle manifest
Use "distributor manifest <command> --help" for command-specific help.
`)
}
func printManifestCreateHelp(w io.Writer) {
fmt.Fprint(w, `Usage:
distributor manifest create <bundle-path> --id <bundle-id> [options]
Options:
--id <bundle-id> Source bundle id
--file <path> Bundle-relative file to include; repeatable
--created <time> RFC3339 source created timestamp
--overwrite Replace an existing manifest.json
--format text|json Output format
Create manifest.json for a local source bundle directory.
`)
}

View File

@@ -33,6 +33,8 @@ func Execute(ctx context.Context, args []string, stdout, stderr io.Writer) int {
return validateCommand(ctx, args[1:], stdout, stderr)
case "inspect":
return inspectCommand(ctx, args[1:], stdout, stderr)
case "manifest":
return manifestCommand(ctx, args[1:], stdout, stderr)
default:
fmt.Fprintf(stderr, "%s: unknown command %q\n\n", app.Name, args[0])
printRootHelp(stderr)
@@ -51,6 +53,7 @@ Commands:
run Run configured distribution pipelines
validate Validate a source bundle or bundle tree
inspect Inspect bundles or distributor state
manifest Create source bundle manifests
Use "%s <command> --help" for command-specific help.
`, app.Name, app.Name, app.Name)

View File

@@ -3,6 +3,9 @@ package cli
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
@@ -10,8 +13,32 @@ import (
"gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
producerbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
)
func decodeEnvelope(t *testing.T, stdout *bytes.Buffer) map[string]any {
t.Helper()
decoder := json.NewDecoder(strings.NewReader(stdout.String()))
var envelope map[string]any
if err := decoder.Decode(&envelope); err != nil {
t.Fatalf("decode JSON envelope: %v; stdout = %q", err, stdout.String())
}
var extra any
if err := decoder.Decode(&extra); err != io.EOF {
t.Fatalf("stdout contains more than one JSON document: %q", stdout.String())
}
return envelope
}
func envelopeResult(t *testing.T, envelope map[string]any) map[string]any {
t.Helper()
result, ok := envelope["result"].(map[string]any)
if !ok {
t.Fatalf("result = %#v, want object", envelope["result"])
}
return result
}
func TestExecuteRootHelp(t *testing.T) {
var stdout, stderr bytes.Buffer
@@ -44,6 +71,43 @@ func TestExecuteVersion(t *testing.T) {
}
}
func TestExecuteVersionJSON(t *testing.T) {
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"version", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
if envelope["command"] != "version" || envelope["ok"] != true {
t.Fatalf("envelope = %#v, want version ok", envelope)
}
result := envelopeResult(t, envelope)
if result["application"] != "distributor" || result["version"] != "dev" {
t.Fatalf("result = %#v, want application/version", result)
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestExecuteRejectsInvalidFormat(t *testing.T) {
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"version", "--format", "xml"}, &stdout, &stderr)
if code != exitUsage {
t.Fatalf("exit code = %d, want %d", code, exitUsage)
}
if stdout.Len() != 0 {
t.Fatalf("stdout = %q, want empty", stdout.String())
}
if !strings.Contains(stderr.String(), "format must be text or json") {
t.Fatalf("stderr = %q, want invalid format error", stderr.String())
}
}
func TestExecuteValidate(t *testing.T) {
var stdout, stderr bytes.Buffer
@@ -57,6 +121,63 @@ func TestExecuteValidate(t *testing.T) {
}
}
func TestExecuteValidateJSON(t *testing.T) {
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"validate", "--format", "json", filepath.Join("..", "bundle", "testdata", "valid_bundle")}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
if envelope["command"] != "validate" || envelope["ok"] != true {
t.Fatalf("envelope = %#v, want validate ok", envelope)
}
result := envelopeResult(t, envelope)
if result["bundle_count"] != float64(1) {
t.Fatalf("result = %#v, want one bundle", result)
}
}
func TestExecuteValidateConfiguredSource(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
configPath := testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot)
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"validate", "--config", configPath, "--pipeline", "reports"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
if got, want := stdout.String(), "Validated 1 bundle(s) for pipeline reports source local\n"; got != want {
t.Fatalf("stdout = %q, want %q", got, want)
}
}
func TestExecuteValidateConfiguredSourceJSON(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "daily", testutil.BundleOptions{ID: "reports.daily"})
configPath := testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot)
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"validate", "--config", configPath, "--pipeline", "reports", "--bundle", "daily", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
if envelope["command"] != "validate" || envelope["ok"] != true {
t.Fatalf("envelope = %#v, want validate ok", envelope)
}
result := envelopeResult(t, envelope)
if result["pipeline_id"] != "reports" || result["source_backend"] != "local" || result["bundle_count"] != float64(1) {
t.Fatalf("result = %#v, want configured source metadata", result)
}
}
func TestExecuteValidateArgs(t *testing.T) {
validPath := filepath.Join("..", "bundle", "testdata", "valid_bundle")
tests := []struct {
@@ -84,6 +205,24 @@ func TestExecuteValidateArgs(t *testing.T) {
wantCode: exitUsage,
wantStderr: "accepts at most one path",
},
{
name: "path plus config",
args: []string{"validate", "--config", "config.yml", "--pipeline", "reports", validPath},
wantCode: exitUsage,
wantStderr: "does not accept a local path",
},
{
name: "pipeline without config",
args: []string{"validate", "--pipeline", "reports"},
wantCode: exitUsage,
wantStderr: "requires --config",
},
{
name: "config without pipeline",
args: []string{"validate", "--config", "config.yml"},
wantCode: exitUsage,
wantStderr: "requires --pipeline",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
@@ -115,6 +254,56 @@ func TestExecuteInspect(t *testing.T) {
}
}
func TestExecuteInspectJSON(t *testing.T) {
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"inspect", "--format", "json", filepath.Join("..", "bundle", "testdata", "valid_bundle")}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
if envelope["command"] != "inspect" || envelope["ok"] != true {
t.Fatalf("envelope = %#v, want inspect ok", envelope)
}
result := envelopeResult(t, envelope)
bundles, ok := result["bundles"].([]any)
if !ok || len(bundles) != 1 {
t.Fatalf("bundles = %#v, want one bundle", result["bundles"])
}
bundle, ok := bundles[0].(map[string]any)
if !ok {
t.Fatalf("bundle = %#v, want object", bundles[0])
}
if bundle["id"] != "weather.daily.brentwood.2026-05-30" || bundle["file_count"] != float64(2) || bundle["total_size"] != float64(24) {
t.Fatalf("bundle = %#v, want normalized metadata", bundle)
}
}
func TestExecuteInspectConfiguredSource(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "daily", testutil.BundleOptions{ID: "reports.daily"})
configPath := testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot)
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"inspect", "--config", configPath, "--pipeline", "reports"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
for _, want := range []string{
"Pipeline: reports",
"Source: local",
"path=daily",
"id=reports.daily",
} {
if !strings.Contains(stdout.String(), want) {
t.Fatalf("stdout = %q, want substring %q", stdout.String(), want)
}
}
}
func TestExecuteInspectArgs(t *testing.T) {
validPath := filepath.Join("..", "bundle", "testdata", "valid_bundle")
tests := []struct {
@@ -142,6 +331,24 @@ func TestExecuteInspectArgs(t *testing.T) {
wantCode: exitUsage,
wantStderr: "accepts at most one path",
},
{
name: "path plus config",
args: []string{"inspect", "--config", "config.yml", "--pipeline", "reports", validPath},
wantCode: exitUsage,
wantStderr: "does not accept a local path",
},
{
name: "pipeline without config",
args: []string{"inspect", "--pipeline", "reports"},
wantCode: exitUsage,
wantStderr: "requires --config",
},
{
name: "config without pipeline",
args: []string{"inspect", "--config", "config.yml"},
wantCode: exitUsage,
wantStderr: "requires --pipeline",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
@@ -160,6 +367,206 @@ func TestExecuteInspectArgs(t *testing.T) {
}
}
func TestExecuteManifestCreateExplicitFiles(t *testing.T) {
root := t.TempDir()
writeCLIFile(t, root, "b.txt", "bravo")
writeCLIFile(t, root, "nested/a.txt", "alpha")
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{
"manifest", "create", root,
"--id", "reports.explicit",
"--created", "2026-06-01T11:00:00Z",
"--file", "b.txt",
"--file", "nested/a.txt",
}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
for _, want := range []string{
"created manifest.json",
"bundle: reports.explicit",
"files: 2",
"digest: sha256:",
} {
if !strings.Contains(stdout.String(), want) {
t.Fatalf("stdout = %q, want substring %q", stdout.String(), want)
}
}
manifest, err := producerbundle.LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if got, want := manifestPaths(manifest), []string{"b.txt", "nested/a.txt"}; !equalStrings(got, want) {
t.Fatalf("manifest paths = %v, want %v", got, want)
}
if err := producerbundle.ValidateBundle(root, manifest); err != nil {
t.Fatalf("ValidateBundle() error = %v", err)
}
var validateStdout, validateStderr bytes.Buffer
validateCode := Execute(context.Background(), []string{"validate", root}, &validateStdout, &validateStderr)
if validateCode != exitOK {
t.Fatalf("validate exit code = %d, want %d; stderr = %q", validateCode, exitOK, validateStderr.String())
}
}
func TestExecuteManifestCreateScansBundle(t *testing.T) {
root := t.TempDir()
writeCLIFile(t, root, "z.txt", "zulu")
writeCLIFile(t, root, ".env", "dotfile")
writeCLIFile(t, root, "nested/report.md", "# Report\n")
writeCLIFile(t, root, storage.StateFileName, "destination state")
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"manifest", "create", root, "--id", "reports.scan"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
manifest, err := producerbundle.LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if got, want := manifestPaths(manifest), []string{".env", "nested/report.md", "z.txt"}; !equalStrings(got, want) {
t.Fatalf("manifest paths = %v, want %v", got, want)
}
}
func TestExecuteManifestCreateJSON(t *testing.T) {
root := t.TempDir()
writeCLIFile(t, root, "report.md", "# Report\n")
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"manifest", "create", root, "--id", "reports.json", "--file", "report.md", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
if envelope["command"] != "manifest create" || envelope["ok"] != true {
t.Fatalf("envelope = %#v, want manifest create ok", envelope)
}
result := envelopeResult(t, envelope)
if result["id"] != "reports.json" || result["file_count"] != float64(1) {
t.Fatalf("result = %#v, want manifest summary", result)
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestExecuteManifestCreateOverwrite(t *testing.T) {
root := t.TempDir()
writeCLIFile(t, root, "report.md", "old\n")
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"manifest", "create", root, "--id", "reports.old", "--file", "report.md"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("initial exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
writeCLIFile(t, root, "report.md", "new\n")
stdout.Reset()
stderr.Reset()
code = Execute(context.Background(), []string{"manifest", "create", root, "--id", "reports.new", "--file", "report.md"}, &stdout, &stderr)
if code != exitError {
t.Fatalf("overwrite exit code = %d, want %d", code, exitError)
}
if !strings.Contains(stderr.String(), "write manifest") {
t.Fatalf("stderr = %q, want write manifest error", stderr.String())
}
manifest, err := producerbundle.LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if manifest.ID != "reports.old" {
t.Fatalf("manifest id = %q, want reports.old", manifest.ID)
}
stdout.Reset()
stderr.Reset()
code = Execute(context.Background(), []string{"manifest", "create", root, "--id", "reports.new", "--file", "report.md", "--overwrite"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("overwrite exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
manifest, err = producerbundle.LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if manifest.ID != "reports.new" {
t.Fatalf("manifest id = %q, want reports.new", manifest.ID)
}
}
func TestExecuteManifestCreateRejectsSymlink(t *testing.T) {
root := t.TempDir()
writeCLIFile(t, root, "target.md", "# Report\n")
if err := os.Symlink("target.md", filepath.Join(root, "link.md")); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"manifest", "create", root, "--id", "reports.link", "--file", "link.md"}, &stdout, &stderr)
if code != exitError {
t.Fatalf("exit code = %d, want %d", code, exitError)
}
if !strings.Contains(stderr.String(), "regular file") {
t.Fatalf("stderr = %q, want regular file error", stderr.String())
}
}
func TestExecuteManifestCreateArgs(t *testing.T) {
root := t.TempDir()
writeCLIFile(t, root, "report.md", "# Report\n")
tests := []struct {
name string
args []string
wantCode int
wantStderr string
}{
{
name: "missing path",
args: []string{"manifest", "create", "--id", "reports.missing"},
wantCode: exitUsage,
wantStderr: "requires exactly one bundle path",
},
{
name: "missing id",
args: []string{"manifest", "create", root},
wantCode: exitError,
wantStderr: "requires --id",
},
{
name: "bad created",
args: []string{"manifest", "create", root, "--id", "reports.bad", "--created", "June 1"},
wantCode: exitError,
wantStderr: "created must be RFC3339",
},
{
name: "bad format",
args: []string{"manifest", "create", root, "--id", "reports.bad", "--format", "xml"},
wantCode: exitUsage,
wantStderr: "format must be text or json",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), tt.args, &stdout, &stderr)
if code != tt.wantCode {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, tt.wantCode, stderr.String())
}
if stdout.Len() != 0 {
t.Fatalf("stdout = %q, want empty", stdout.String())
}
if !strings.Contains(stderr.String(), tt.wantStderr) {
t.Fatalf("stderr = %q, want substring %q", stderr.String(), tt.wantStderr)
}
})
}
}
func TestExecuteRunDryRun(t *testing.T) {
sourceRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
@@ -180,6 +587,269 @@ func TestExecuteRunDryRun(t *testing.T) {
}
}
func TestExecuteRunJSONDryRun(t *testing.T) {
sourceRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
configPath := testutil.WriteMinimalLocalConfig(t, sourceRoot, t.TempDir())
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"run", "--config", configPath, "--dry-run", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
if envelope["command"] != "run" || envelope["ok"] != true {
t.Fatalf("envelope = %#v, want run ok", envelope)
}
result := envelopeResult(t, envelope)
if result["dry_run"] != true {
t.Fatalf("result = %#v, want dry_run true", result)
}
actions, ok := result["actions"].([]any)
if !ok || len(actions) != 1 {
t.Fatalf("actions = %#v, want one action", result["actions"])
}
action, ok := actions[0].(map[string]any)
if !ok || action["action"] != "publish_new" {
t.Fatalf("action = %#v, want publish_new", actions[0])
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestExecuteRunJSONDryRunReportsFixedPathMapping(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(`
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: latest
backend: local
path: `+destinationRoot+`
path_mapping:
mode: fixed
`), 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"run", "--config", configPath, "--dry-run", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
warnings, ok := envelope["warnings"].([]any)
if !ok || len(warnings) != 1 {
t.Fatalf("warnings = %#v, want one fixed-path warning", envelope["warnings"])
}
warning, ok := warnings[0].(map[string]any)
if !ok || !strings.Contains(fmt.Sprint(warning["message"]), "path_mapping=fixed candidates=1 selected_bundle=.") {
t.Fatalf("warning = %#v, want fixed-path selection warning", warnings[0])
}
result := envelopeResult(t, envelope)
actions, ok := result["actions"].([]any)
if !ok || len(actions) != 1 {
t.Fatalf("actions = %#v, want one action", result["actions"])
}
action, ok := actions[0].(map[string]any)
if !ok || action["path_mapping"] != "fixed" || action["destination_path"] != "." || action["action"] != "publish_new" {
t.Fatalf("action = %#v, want fixed publish_new at root", actions[0])
}
summary, ok := result["summary"].(map[string]any)
if !ok || summary["fixed_path"] != float64(1) {
t.Fatalf("summary = %#v, want fixed_path 1", result["summary"])
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestExecuteRunJSONDryRunReportsLinks(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(`
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: web
backend: local
path: `+destinationRoot+`
links:
base_url: https://reports.example.com/archive
primary: source
`), 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"run", "--config", configPath, "--dry-run", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
result := envelopeResult(t, envelope)
actions, ok := result["actions"].([]any)
if !ok || len(actions) != 1 {
t.Fatalf("actions = %#v, want one action", result["actions"])
}
action, ok := actions[0].(map[string]any)
if !ok || action["primary_url"] != "https://reports.example.com/archive/report.md" {
t.Fatalf("action = %#v, want primary URL", actions[0])
}
outputs, ok := action["outputs"].([]any)
if !ok || len(outputs) != 2 {
t.Fatalf("outputs = %#v, want two outputs", action["outputs"])
}
output, ok := outputs[0].(map[string]any)
if !ok || output["url"] != "https://reports.example.com/archive/report.md" {
t.Fatalf("output = %#v, want output URL", outputs[0])
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestExecuteRunJSONWarningsAreStructured(t *testing.T) {
name := "DISTRIBUTOR_TEST_CLI_JSON_SECRET"
t.Setenv(name, "process-value")
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
secretsRoot := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
if err := os.WriteFile(filepath.Join(secretsRoot, name), []byte("secret-value\n"), 0o600); err != nil {
t.Fatalf("write secret: %v", err)
}
configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(`
secrets:
directory: `+secretsRoot+`
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
`), 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"run", "--config", configPath, "--dry-run", "--format", "json"}, &stdout, &stderr)
if code != exitOK {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
warnings, ok := envelope["warnings"].([]any)
if !ok || len(warnings) != 1 {
t.Fatalf("warnings = %#v, want one warning", envelope["warnings"])
}
warning, ok := warnings[0].(map[string]any)
if !ok || !strings.Contains(fmt.Sprint(warning["message"]), name) {
t.Fatalf("warning = %#v, want secret name", warnings[0])
}
if strings.Contains(stdout.String(), "Warning:") || strings.Contains(stdout.String(), "process-value") || strings.Contains(stdout.String(), "secret-value") {
t.Fatalf("stdout exposed text warning or secret values: %q", stdout.String())
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestExecuteRunJSONFatalSetupErrorWritesNoJSON(t *testing.T) {
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"run", "--config", filepath.Join(t.TempDir(), "missing.yml"), "--format", "json"}, &stdout, &stderr)
if code != exitError {
t.Fatalf("exit code = %d, want %d", code, exitError)
}
if stdout.Len() != 0 {
t.Fatalf("stdout = %q, want empty", stdout.String())
}
if !strings.Contains(stderr.String(), "no such file or directory") {
t.Fatalf("stderr = %q, want setup error", stderr.String())
}
}
func TestExecuteRunJSONPartialFailure(t *testing.T) {
sourceRoot := t.TempDir()
firstDestination := t.TempDir()
secondDestination := t.TempDir()
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
if err := os.WriteFile(filepath.Join(firstDestination, "unmanaged.txt"), []byte("data"), 0o600); err != nil {
t.Fatalf("write unmanaged file: %v", err)
}
configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(`
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive-one
backend: local
path: `+firstDestination+`
- id: archive-two
backend: local
path: `+secondDestination+`
`), 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
var stdout, stderr bytes.Buffer
code := Execute(context.Background(), []string{"run", "--config", configPath, "--format", "json"}, &stdout, &stderr)
if code != exitError {
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitError, stderr.String())
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty for partial JSON result", stderr.String())
}
envelope := decodeEnvelope(t, &stdout)
if envelope["command"] != "run" || envelope["ok"] != false {
t.Fatalf("envelope = %#v, want failed run envelope", envelope)
}
errors, ok := envelope["errors"].([]any)
if !ok || len(errors) != 1 {
t.Fatalf("errors = %#v, want one error", envelope["errors"])
}
result := envelopeResult(t, envelope)
summary, ok := result["summary"].(map[string]any)
if !ok || summary["status"] != "failed" || summary["failed"] != float64(1) {
t.Fatalf("summary = %#v, want failed summary", result["summary"])
}
actions, ok := result["actions"].([]any)
if !ok || len(actions) != 2 {
t.Fatalf("actions = %#v, want two actions", result["actions"])
}
if _, err := os.Stat(filepath.Join(secondDestination, storage.StateFileName)); err != nil {
t.Fatalf("second destination state stat error = %v", err)
}
}
func TestExecuteRunForceDryRunReportsWithoutWriting(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
@@ -249,3 +919,34 @@ func TestUnknownCommandIsUsageError(t *testing.T) {
t.Fatalf("stderr = %q, want unknown command error", stderr.String())
}
}
func manifestPaths(manifest producerbundle.Manifest) []string {
paths := make([]string, 0, len(manifest.Files))
for _, file := range manifest.Files {
paths = append(paths, file.Path)
}
return paths
}
func equalStrings(a, b []string) bool {
if len(a) != len(b) {
return false
}
for index := range a {
if a[index] != b[index] {
return false
}
}
return true
}
func writeCLIFile(t *testing.T, root, relative, body string) {
t.Helper()
path := filepath.Join(root, filepath.FromSlash(relative))
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("mkdir %s: %v", filepath.Dir(path), err)
}
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatalf("write %s: %v", path, err)
}
}

View File

@@ -20,19 +20,28 @@ func runCommand(ctx context.Context, args []string, stdout, stderr io.Writer) in
configPath := flags.String("config", "", "path to config file")
dryRun := flags.Bool("dry-run", false, "load and validate config without publishing")
force := flags.Bool("force", false, "allow explicit destructive replacement for supported conflicts")
formatFlag := addFormatFlag(flags)
if err := flags.Parse(args); err != nil {
return exitUsage
}
if rejectPositionalArgs(stderr, "run", flags.Args()) {
return exitUsage
}
format, ok := parseOutputFormat(stderr, "run", *formatFlag)
if !ok {
return exitUsage
}
if err := app.Run(ctx, app.RunOptions{
ConfigPath: *configPath,
DryRun: *dryRun,
Force: *force,
Stdout: stdout,
ConfigPath: *configPath,
DryRun: *dryRun,
Force: *force,
Stdout: stdout,
OutputFormat: format,
}); err != nil {
if app.IsJSONOutput(format) && app.IsPartialResultError(err) {
return exitError
}
return fail(stderr, err)
}
return exitOK
@@ -40,12 +49,14 @@ func runCommand(ctx context.Context, args []string, stdout, stderr io.Writer) in
func printRunHelp(w io.Writer) {
fmt.Fprint(w, `Usage:
distributor run --config <path> [--dry-run] [--force]
distributor run --config <path> [--dry-run] [--force] [--format text|json]
Options:
--config <path> Path to config file
--dry-run Load and validate config without publishing
--force Allow explicit destructive replacement for supported conflicts
--format text|json
Output format
Run discovers configured source bundles, plans each destination, publishes
selected outputs unless --dry-run is set, and prints a final status summary.

View File

@@ -0,0 +1,26 @@
package cli
import (
"fmt"
"io"
)
func validateInspectModeOK(stderr io.Writer, command, path, configPath, pipelineID, bundlePath string) bool {
configMode := configPath != "" || pipelineID != "" || bundlePath != ""
if !configMode {
return true
}
if path != "" {
fmt.Fprintf(stderr, "distributor: %s does not accept a local path with --config, --pipeline, or --bundle\n", command)
return false
}
if configPath == "" {
fmt.Fprintf(stderr, "distributor: %s requires --config when --pipeline or --bundle is set\n", command)
return false
}
if pipelineID == "" {
fmt.Fprintf(stderr, "distributor: %s requires --pipeline in config mode\n", command)
return false
}
return true
}

View File

@@ -2,6 +2,7 @@ package cli
import (
"context"
"flag"
"fmt"
"io"
@@ -13,11 +14,34 @@ func validateCommand(ctx context.Context, args []string, stdout, stderr io.Write
printValidateHelp(stdout)
return exitOK
}
path, ok := parseOptionalPathArg(stderr, "validate", args)
flags := flag.NewFlagSet("validate", flag.ContinueOnError)
flags.SetOutput(stderr)
configPath := flags.String("config", "", "path to config file")
pipelineID := flags.String("pipeline", "", "pipeline id")
bundlePath := flags.String("bundle", "", "source-root-relative bundle path")
formatFlag := addFormatFlag(flags)
if err := flags.Parse(args); err != nil {
return exitUsage
}
format, ok := parseOutputFormat(stderr, "validate", *formatFlag)
if !ok {
return exitUsage
}
if err := app.Validate(ctx, app.ValidateOptions{Path: path, Stdout: stdout}); err != nil {
path, ok := parseOptionalPathArg(stderr, "validate", flags.Args())
if !ok {
return exitUsage
}
if !validateInspectModeOK(stderr, "validate", path, *configPath, *pipelineID, *bundlePath) {
return exitUsage
}
if err := app.Validate(ctx, app.ValidateOptions{
Path: path,
ConfigPath: *configPath,
PipelineID: *pipelineID,
BundlePath: *bundlePath,
Stdout: stdout,
OutputFormat: format,
}); err != nil {
return fail(stderr, err)
}
return exitOK
@@ -25,8 +49,16 @@ func validateCommand(ctx context.Context, args []string, stdout, stderr io.Write
func printValidateHelp(w io.Writer) {
fmt.Fprint(w, `Usage:
distributor validate <path>
distributor validate [--format text|json] <path>
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
Validate a local source bundle directory or a tree containing source bundles.
Options:
--config <path> Path to config file for configured source validation
--pipeline <id> Pipeline id to validate in config mode
--bundle <path> Source-root-relative bundle path to validate
--format text|json Output format
Validate a local source bundle directory, a local source bundle tree, or a
configured pipeline source.
`)
}

View File

@@ -2,6 +2,7 @@ package cli
import (
"context"
"flag"
"fmt"
"io"
@@ -13,16 +14,44 @@ func versionCommand(_ context.Context, args []string, stdout, stderr io.Writer)
printVersionHelp(stdout)
return exitOK
}
if rejectExtraArgs(stderr, "version", args) {
flags := flag.NewFlagSet("version", flag.ContinueOnError)
flags.SetOutput(stderr)
formatFlag := addFormatFlag(flags)
if err := flags.Parse(args); err != nil {
return exitUsage
}
if rejectExtraArgs(stderr, "version", flags.Args()) {
return exitUsage
}
format, ok := parseOutputFormat(stderr, "version", *formatFlag)
if !ok {
return exitUsage
}
if app.IsJSONOutput(format) {
err := app.WriteJSONEnvelope(stdout, "version", true, nil, versionResult{
Application: app.Name,
Version: app.Version,
}, nil)
if err != nil {
return fail(stderr, err)
}
return exitOK
}
fmt.Fprintln(stdout, app.VersionString())
return exitOK
}
type versionResult struct {
Application string `json:"application"`
Version string `json:"version"`
}
func printVersionHelp(w io.Writer) {
fmt.Fprint(w, `Usage:
distributor version
distributor version [--format text|json]
Options:
--format text|json Output format
Print version information.
`)

View File

@@ -32,6 +32,8 @@ type Destination struct {
SSH SSH `yaml:",inline"`
Publish *PublishPolicy `yaml:"publish"`
Transform Transform `yaml:"transform"`
PathMap PathMapping `yaml:"path_mapping"`
Links *Links `yaml:"links"`
Transfer TransferPolicy `yaml:"transfer"`
}
@@ -77,6 +79,16 @@ type Transform struct {
type MarkdownToHTML struct {
Enabled bool `yaml:"enabled"`
Mode string `yaml:"mode"`
Input string `yaml:"input"`
}
type PathMapping struct {
Mode string `yaml:"mode"`
}
type Links struct {
BaseURL string `yaml:"base_url"`
Primary string `yaml:"primary"`
}
type TransferPolicy struct {

View File

@@ -1,5 +1,7 @@
package config
import "gitea.maximumdirect.net/eric/distributor/internal/transform"
const DefaultConfigPath = "/usr/local/etc/distributor/config.yml"
const (
@@ -19,7 +21,19 @@ const (
)
const (
TransformModeSidecar = "sidecar"
TransformModeSidecar = transform.MarkdownModeSidecar
TransformModeIndex = transform.MarkdownModeIndex
)
const (
PathMappingPreserveRelative = "preserve_relative"
PathMappingFixed = "fixed"
)
const (
LinkPrimaryAuto = "auto"
LinkPrimaryHTML = "html"
LinkPrimarySource = "source"
)
const DefaultS3Region = "us-east-1"
@@ -37,6 +51,15 @@ func ApplyDefaults(cfg *Config) {
if destination.Publish == nil {
destination.Publish = &PublishPolicy{Source: true}
}
if destination.Transform.MarkdownToHTML != nil && destination.Transform.MarkdownToHTML.Mode == "" {
destination.Transform.MarkdownToHTML.Mode = TransformModeSidecar
}
if destination.PathMap.Mode == "" {
destination.PathMap.Mode = PathMappingPreserveRelative
}
if destination.Links != nil && destination.Links.Primary == "" {
destination.Links.Primary = LinkPrimaryAuto
}
if destination.Transfer.OnDestinationSame == "" {
destination.Transfer.OnDestinationSame = TransferActionSkip
}

View File

@@ -94,6 +94,120 @@ pipelines:
}
}
func TestLoadFileValidMarkdownIndexConfig(t *testing.T) {
cfg := loadConfig(t, `
pipelines:
- id: static-site
source:
backend: local
path: /var/spool/reports
destinations:
- id: web
backend: local
path: /srv/www/reports
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
`)
markdown := cfg.Pipelines[0].Destinations[0].Transform.MarkdownToHTML
if markdown == nil || markdown.Mode != TransformModeIndex || markdown.Input != "report.md" {
t.Fatalf("markdown config = %#v, want index input", markdown)
}
}
func TestLoadFileDefaultsMarkdownModeToSidecar(t *testing.T) {
cfg := loadConfig(t, `
pipelines:
- id: static-site
source:
backend: local
path: /var/spool/reports
destinations:
- id: web
backend: local
path: /srv/www/reports
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
`)
markdown := cfg.Pipelines[0].Destinations[0].Transform.MarkdownToHTML
if markdown == nil || markdown.Mode != TransformModeSidecar {
t.Fatalf("markdown mode = %#v, want sidecar default", markdown)
}
}
func TestLoadFileDefaultsPathMappingToPreserveRelative(t *testing.T) {
cfg := loadConfig(t, `
pipelines:
- id: reports
source:
backend: local
path: /source
destinations:
- id: archive
backend: local
path: /destination
`)
if got, want := cfg.Pipelines[0].Destinations[0].PathMap.Mode, PathMappingPreserveRelative; got != want {
t.Fatalf("path mapping mode = %q, want %q", got, want)
}
}
func TestLoadFileAcceptsFixedPathMapping(t *testing.T) {
cfg := loadConfig(t, `
pipelines:
- id: reports
source:
backend: local
path: /source
destinations:
- id: latest
backend: local
path: /destination/latest
path_mapping:
mode: fixed
`)
if got, want := cfg.Pipelines[0].Destinations[0].PathMap.Mode, PathMappingFixed; got != want {
t.Fatalf("path mapping mode = %q, want %q", got, want)
}
}
func TestLoadFileDefaultsLinksPrimaryToAuto(t *testing.T) {
cfg := loadConfig(t, `
pipelines:
- id: reports
source:
backend: local
path: /source
destinations:
- id: web
backend: local
path: /destination
links:
base_url: https://reports.example.com/archive
`)
links := cfg.Pipelines[0].Destinations[0].Links
if links == nil {
t.Fatal("links = nil, want config")
}
if links.BaseURL != "https://reports.example.com/archive" || links.Primary != LinkPrimaryAuto {
t.Fatalf("links = %#v, want base URL with auto primary", links)
}
}
func TestLoadFileValidBackendConfigs(t *testing.T) {
tests := map[string]string{
"local": `
@@ -464,7 +578,9 @@ func TestExampleConfigsLoad(t *testing.T) {
"../../examples/local-to-local.yml",
"../../examples/local-publish.yml",
"../../examples/local-html.yml",
"../../examples/local-index.yml",
"../../examples/fan-out.yml",
"../../examples/archive-and-latest.yml",
"../../examples/ssh-destination.yml",
"../../examples/s3-destination.yml",
} {

View File

@@ -2,6 +2,7 @@ package config
import (
"fmt"
"net/url"
"regexp"
"strings"
)
@@ -58,6 +59,8 @@ func Validate(cfg Config) error {
errs = validateDestinationBackend(errs, destinationContext, destination)
errs = validatePublishTransformPolicy(errs, destinationContext, destination.Publish, destination.Transform)
errs = validatePathMapping(errs, destinationContext+".path_mapping", destination.PathMap)
errs = validateLinks(errs, destinationContext+".links", destination.Links)
errs = validateTransferPolicy(errs, destinationContext+".transfer", destination.Transfer)
}
}
@@ -149,17 +152,68 @@ func ValidatePublishTransformPolicy(publish PublishPolicy, transform Transform)
if transform.MarkdownToHTML == nil {
return nil
}
mode := transform.MarkdownToHTML.Mode
if mode == "" {
mode = TransformModeSidecar
}
if mode != TransformModeSidecar && mode != TransformModeIndex {
return fmt.Errorf("transform.markdown_to_html.mode must be %s or %s", TransformModeSidecar, TransformModeIndex)
}
if transform.MarkdownToHTML.Input != "" && !transform.MarkdownToHTML.Enabled {
return fmt.Errorf("transform.markdown_to_html.input requires transform.markdown_to_html.enabled to be true")
}
if transform.MarkdownToHTML.Input != "" && mode != TransformModeIndex {
return fmt.Errorf("transform.markdown_to_html.input is only valid when mode is %s", TransformModeIndex)
}
if transform.MarkdownToHTML.Enabled && !publish.HTML {
return fmt.Errorf("transform.markdown_to_html.enabled requires publish.html to be true")
}
if publish.HTML && !transform.MarkdownToHTML.Enabled {
return fmt.Errorf("transform.markdown_to_html.enabled must be true when publish.html is true")
}
if publish.HTML && transform.MarkdownToHTML.Mode != TransformModeSidecar {
return fmt.Errorf("transform.markdown_to_html.mode must be %s", TransformModeSidecar)
return nil
}
func validatePathMapping(errs ValidationErrors, context string, mapping PathMapping) ValidationErrors {
if mapping.Mode != PathMappingPreserveRelative && mapping.Mode != PathMappingFixed {
errs = append(errs, context+".mode must be "+PathMappingPreserveRelative+" or "+PathMappingFixed)
}
if transform.MarkdownToHTML.Enabled && transform.MarkdownToHTML.Mode != TransformModeSidecar {
return fmt.Errorf("transform.markdown_to_html.mode must be %s", TransformModeSidecar)
return errs
}
func validateLinks(errs ValidationErrors, context string, links *Links) ValidationErrors {
if links == nil {
return errs
}
if !transform.MarkdownToHTML.Enabled && transform.MarkdownToHTML.Mode != "" && transform.MarkdownToHTML.Mode != TransformModeSidecar {
return fmt.Errorf("transform.markdown_to_html.mode must be %s", TransformModeSidecar)
if links.BaseURL == "" {
errs = append(errs, context+".base_url is required")
} else if err := validateLinkBaseURL(links.BaseURL); err != nil {
errs = append(errs, context+".base_url "+err.Error())
}
switch links.Primary {
case LinkPrimaryAuto, LinkPrimaryHTML, LinkPrimarySource:
default:
errs = append(errs, context+".primary must be "+LinkPrimaryAuto+", "+LinkPrimaryHTML+", or "+LinkPrimarySource)
}
return errs
}
func validateLinkBaseURL(value string) error {
parsed, err := url.Parse(value)
if err != nil {
return fmt.Errorf("must be a valid URL")
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("must use http or https")
}
if parsed.Host == "" {
return fmt.Errorf("must include a host")
}
if parsed.RawQuery != "" {
return fmt.Errorf("must not include a query string")
}
if parsed.Fragment != "" {
return fmt.Errorf("must not include a fragment")
}
return nil
}

View File

@@ -70,6 +70,81 @@ func TestValidateAcceptsForceReplacementTransferActions(t *testing.T) {
}
}
func TestValidatePathMapping(t *testing.T) {
tests := []struct {
name string
mode string
wantErr bool
}{
{name: "preserve relative", mode: PathMappingPreserveRelative},
{name: "fixed", mode: PathMappingFixed},
{name: "invalid", mode: "archive", wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Config{Pipelines: []Pipeline{{
ID: "reports",
Source: Backend{Backend: BackendLocal, Path: "/source"},
Destinations: []Destination{{
ID: "archive",
Backend: BackendLocal,
Path: "/destination",
PathMap: PathMapping{Mode: tt.mode},
}},
}}}
ApplyDefaults(&cfg)
err := Validate(cfg)
if tt.wantErr && err == nil {
t.Fatal("Validate() error = nil, want error")
}
if !tt.wantErr && err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
func TestValidateLinks(t *testing.T) {
tests := []struct {
name string
links *Links
wantErr bool
}{
{name: "absent links"},
{name: "http", links: &Links{BaseURL: "http://reports.example.com/archive", Primary: LinkPrimaryAuto}},
{name: "https", links: &Links{BaseURL: "https://reports.example.com/archive/", Primary: LinkPrimaryHTML}},
{name: "source primary", links: &Links{BaseURL: "https://reports.example.com", Primary: LinkPrimarySource}},
{name: "missing base", links: &Links{Primary: LinkPrimaryAuto}, wantErr: true},
{name: "ftp scheme", links: &Links{BaseURL: "ftp://reports.example.com", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "missing host", links: &Links{BaseURL: "https:///archive", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "query", links: &Links{BaseURL: "https://reports.example.com/archive?preview=1", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "fragment", links: &Links{BaseURL: "https://reports.example.com/archive#top", Primary: LinkPrimaryAuto}, wantErr: true},
{name: "invalid primary", links: &Links{BaseURL: "https://reports.example.com", Primary: "document"}, wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Config{Pipelines: []Pipeline{{
ID: "reports",
Source: Backend{Backend: BackendLocal, Path: "/source"},
Destinations: []Destination{{
ID: "web",
Backend: BackendLocal,
Path: "/destination",
Links: tt.links,
}},
}}}
ApplyDefaults(&cfg)
err := Validate(cfg)
if tt.wantErr && err == nil {
t.Fatal("Validate() error = nil, want error")
}
if !tt.wantErr && err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
type publishTransformPolicyCase struct {
name string
publish PublishPolicy
@@ -91,6 +166,30 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
Mode: TransformModeSidecar,
}},
},
{
name: "html only default mode allowed",
publish: PublishPolicy{HTML: true},
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
Enabled: true,
}},
},
{
name: "html only index allowed",
publish: PublishPolicy{HTML: true},
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
Enabled: true,
Mode: TransformModeIndex,
}},
},
{
name: "html only index input allowed",
publish: PublishPolicy{HTML: true},
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
Enabled: true,
Mode: TransformModeIndex,
Input: "report.md",
}},
},
{
name: "source and html sidecar allowed",
publish: PublishPolicy{Source: true, HTML: true},
@@ -127,6 +226,15 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
}},
wantErr: true,
},
{
name: "source only enabled transform rejected",
publish: PublishPolicy{Source: true},
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
Enabled: true,
Mode: TransformModeSidecar,
}},
wantErr: true,
},
{
name: "enabled markdown wrong mode rejected",
publish: PublishPolicy{Source: true},
@@ -136,6 +244,16 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
}},
wantErr: true,
},
{
name: "sidecar input rejected",
publish: PublishPolicy{HTML: true},
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
Enabled: true,
Mode: TransformModeSidecar,
Input: "report.md",
}},
wantErr: true,
},
{
name: "disabled markdown empty mode allowed",
publish: PublishPolicy{Source: true},
@@ -151,6 +269,24 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
Mode: TransformModeSidecar,
}},
},
{
name: "disabled markdown index mode allowed",
publish: PublishPolicy{Source: true},
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
Enabled: false,
Mode: TransformModeIndex,
}},
},
{
name: "disabled markdown input rejected",
publish: PublishPolicy{Source: true},
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
Enabled: false,
Mode: TransformModeIndex,
Input: "report.md",
}},
wantErr: true,
},
{
name: "disabled markdown wrong mode rejected",
publish: PublishPolicy{Source: true},

View File

@@ -78,6 +78,9 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
Source: state.SourceState{Manifest: req.SourceBundle.Manifest},
Outputs: stateOutputs(plan.Outputs),
}
if plan.PrimaryURL != "" {
destinationState.Links = &state.LinkState{PrimaryURL: plan.PrimaryURL}
}
if err := state.Validate(destinationState); err != nil {
cleanup()
return err

116
internal/publish/links.go Normal file
View File

@@ -0,0 +1,116 @@
package publish
import (
"fmt"
"net/url"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/state"
)
func PlanLinks(req Request, outputs []Output) ([]Output, string, error) {
if req.Links == nil {
return outputs, "", nil
}
linked := make([]Output, 0, len(outputs))
for _, output := range outputs {
outputURL, err := OutputURL(req.Links.BaseURL, req.DestinationBundlePath, output.DestinationPath)
if err != nil {
return nil, "", err
}
output.URL = outputURL
linked = append(linked, output)
}
return linked, primaryURL(linked, req.Links.Primary), nil
}
func OutputURL(baseURL, destinationBundlePath, outputPath string) (string, error) {
parsed, err := url.Parse(baseURL)
if err != nil {
return "", fmt.Errorf("link base URL: %w", err)
}
segments := pathSegments(parsed.Path)
segments = append(segments, pathSegments(destinationBundlePath)...)
outputSegments := pathSegments(outputPath)
trailingSlash := false
if len(outputSegments) > 0 && outputSegments[len(outputSegments)-1] == "index.html" {
outputSegments = outputSegments[:len(outputSegments)-1]
trailingSlash = true
}
segments = append(segments, outputSegments...)
parsed.Path = urlPath(segments, trailingSlash)
parsed.RawPath = ""
return parsed.String(), nil
}
func pathSegments(value string) []string {
trimmed := strings.Trim(value, "/")
if trimmed == "" {
return nil
}
return strings.Split(trimmed, "/")
}
func urlPath(segments []string, trailingSlash bool) string {
if len(segments) == 0 {
return "/"
}
path := "/" + strings.Join(segments, "/")
if trailingSlash && !strings.HasSuffix(path, "/") {
path += "/"
}
return path
}
func primaryURL(outputs []Output, policy string) string {
switch policy {
case config.LinkPrimaryHTML:
return firstGeneratedHTMLURL(outputs)
case config.LinkPrimarySource:
return firstSourceURL(outputs)
default:
if url := firstIndexURL(outputs); url != "" {
return url
}
if url := firstGeneratedHTMLURL(outputs); url != "" {
return url
}
return firstSourceURL(outputs)
}
}
func firstIndexURL(outputs []Output) string {
for _, output := range outputs {
if isIndexOutput(output.DestinationPath) {
return output.URL
}
}
return ""
}
func firstGeneratedHTMLURL(outputs []Output) string {
for _, output := range outputs {
if output.Kind == state.OutputKindGenerated && isHTMLOutput(output.DestinationPath) {
return output.URL
}
}
return ""
}
func firstSourceURL(outputs []Output) string {
for _, output := range outputs {
if output.Kind == state.OutputKindSource {
return output.URL
}
}
return ""
}
func isIndexOutput(path string) bool {
return path == "index.html" || strings.HasSuffix(path, "/index.html")
}
func isHTMLOutput(path string) bool {
return strings.HasSuffix(path, ".html")
}

View File

@@ -0,0 +1,149 @@
package publish
import (
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/state"
)
func TestOutputURLUsesURLPathSemantics(t *testing.T) {
tests := []struct {
name string
baseURL string
destinationBundlePath string
outputPath string
want string
}{
{
name: "nested non index",
baseURL: "https://reports.example.com/archive",
destinationBundlePath: "daily/brentwood",
outputPath: "report.html",
want: "https://reports.example.com/archive/daily/brentwood/report.html",
},
{
name: "nested index",
baseURL: "https://reports.example.com/archive",
destinationBundlePath: "daily/brentwood",
outputPath: "index.html",
want: "https://reports.example.com/archive/daily/brentwood/",
},
{
name: "fixed index",
baseURL: "https://reports.example.com/latest",
destinationBundlePath: "",
outputPath: "index.html",
want: "https://reports.example.com/latest/",
},
{
name: "escaped segments",
baseURL: "https://reports.example.com/archive root",
destinationBundlePath: "daily reports",
outputPath: "morning report.html",
want: "https://reports.example.com/archive%20root/daily%20reports/morning%20report.html",
},
{
name: "nested output index",
baseURL: "https://reports.example.com",
destinationBundlePath: "daily",
outputPath: "site/index.html",
want: "https://reports.example.com/daily/site/",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := OutputURL(tt.baseURL, tt.destinationBundlePath, tt.outputPath)
if err != nil {
t.Fatalf("OutputURL() error = %v", err)
}
if got != tt.want {
t.Fatalf("OutputURL() = %q, want %q", got, tt.want)
}
})
}
}
func TestPlanLinksSelectsPrimaryURL(t *testing.T) {
outputs := []Output{
{
DestinationPath: "report.md",
Kind: state.OutputKindSource,
},
{
DestinationPath: "report.html",
Kind: state.OutputKindGenerated,
},
{
DestinationPath: "index.html",
Kind: state.OutputKindGenerated,
},
}
tests := []struct {
name string
primary string
want string
}{
{name: "auto prefers index", primary: config.LinkPrimaryAuto, want: "https://reports.example.com/daily/"},
{name: "html uses first generated html", primary: config.LinkPrimaryHTML, want: "https://reports.example.com/daily/report.html"},
{name: "source uses first source", primary: config.LinkPrimarySource, want: "https://reports.example.com/daily/report.md"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
linked, primaryURL, err := PlanLinks(Request{
DestinationBundlePath: "daily",
Links: &config.Links{
BaseURL: "https://reports.example.com",
Primary: tt.primary,
},
}, outputs)
if err != nil {
t.Fatalf("PlanLinks() error = %v", err)
}
if primaryURL != tt.want {
t.Fatalf("primary URL = %q, want %q", primaryURL, tt.want)
}
for index, output := range linked {
if output.URL == "" {
t.Fatalf("linked output %d has empty URL", index)
}
}
})
}
}
func TestPlanLinksReturnsNoPrimaryWhenPolicyHasNoMatch(t *testing.T) {
linked, primaryURL, err := PlanLinks(Request{
DestinationBundlePath: "daily",
Links: &config.Links{
BaseURL: "https://reports.example.com",
Primary: config.LinkPrimaryHTML,
},
}, []Output{{
DestinationPath: "report.md",
Kind: state.OutputKindSource,
}})
if err != nil {
t.Fatalf("PlanLinks() error = %v", err)
}
if primaryURL != "" {
t.Fatalf("primary URL = %q, want empty", primaryURL)
}
if linked[0].URL != "https://reports.example.com/daily/report.md" {
t.Fatalf("linked URL = %q", linked[0].URL)
}
}
func TestPlanLinksLeavesOutputsUnchangedWithoutConfig(t *testing.T) {
outputs := []Output{{DestinationPath: "report.md", Kind: state.OutputKindSource}}
linked, primaryURL, err := PlanLinks(Request{}, outputs)
if err != nil {
t.Fatalf("PlanLinks() error = %v", err)
}
if primaryURL != "" {
t.Fatalf("primary URL = %q, want empty", primaryURL)
}
if linked[0].URL != "" {
t.Fatalf("output URL = %q, want empty", linked[0].URL)
}
}

View File

@@ -26,6 +26,10 @@ func PlanOutputs(ctx context.Context, req Request) ([]Output, error) {
generatedOutputs, err := transformer.Generate(ctx, transform.Request{
SourceBundle: req.SourceBundle,
SourceBackend: req.SourceBackend,
Markdown: transform.MarkdownOptions{
Mode: req.Transform.MarkdownToHTML.Mode,
Input: req.Transform.MarkdownToHTML.Input,
},
})
if err != nil {
return nil, err
@@ -101,6 +105,7 @@ func stateOutputs(outputs []Output) []state.OutputFile {
Kind: output.Kind,
SourcePath: output.SourcePath,
Transform: output.Transform,
URL: output.URL,
SHA256: output.SHA256,
Size: output.Size,
})

View File

@@ -103,6 +103,35 @@ func TestPlanOutputsUsesRegisteredTransformer(t *testing.T) {
}
}
func TestPlanOutputsPassesMarkdownOptions(t *testing.T) {
data := []byte("<p>Generated</p>\n")
transformer := &recordingTransformer{outputs: []transform.Output{{
Path: "index.html",
SourcePath: "report.md",
Transform: transform.MarkdownToHTML,
Data: data,
SHA256: bundle.FileDigest(data),
Size: int64(len(data)),
}}}
_, err := PlanOutputs(context.Background(), Request{
Publish: config.PublishPolicy{HTML: true},
Transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{
Enabled: true,
Mode: config.TransformModeIndex,
Input: "report.md",
}},
Transformers: testResolver{transform.MarkdownToHTML: transformer},
})
if err != nil {
t.Fatalf("PlanOutputs() error = %v", err)
}
if transformer.request.Markdown.Mode != config.TransformModeIndex || transformer.request.Markdown.Input != "report.md" {
t.Fatalf("markdown options = %#v, want index/report.md", transformer.request.Markdown)
}
}
func TestBuildRejectsHTMLWithoutTransform(t *testing.T) {
sourceBackend := fake.New()
destinationBackend := fake.New()
@@ -148,6 +177,14 @@ func TestValidateRequestChecksPublishTransformPolicy(t *testing.T) {
Mode: config.TransformModeSidecar,
}},
},
{
name: "html only index allowed",
publish: config.PublishPolicy{HTML: true},
transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{
Enabled: true,
Mode: config.TransformModeIndex,
}},
},
{
name: "source and html sidecar allowed",
publish: config.PublishPolicy{Source: true, HTML: true},
@@ -184,6 +221,15 @@ func TestValidateRequestChecksPublishTransformPolicy(t *testing.T) {
}},
wantErr: true,
},
{
name: "source only enabled transform rejected",
publish: config.PublishPolicy{Source: true},
transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{
Enabled: true,
Mode: config.TransformModeSidecar,
}},
wantErr: true,
},
{
name: "enabled markdown wrong mode rejected",
publish: config.PublishPolicy{Source: true},
@@ -253,3 +299,14 @@ type testTransformer struct {
func (t testTransformer) Generate(context.Context, transform.Request) ([]transform.Output, error) {
return t.outputs, t.err
}
type recordingTransformer struct {
outputs []transform.Output
request transform.Request
err error
}
func (t *recordingTransformer) Generate(_ context.Context, req transform.Request) ([]transform.Output, error) {
t.request = req
return t.outputs, t.err
}

View File

@@ -30,8 +30,10 @@ type Request struct {
SourceBackend storage.Backend
DestinationBackend storage.Backend
DestinationBundlePath string
PathMapping string
Publish config.PublishPolicy
Transform config.Transform
Links *config.Links
Transformers TransformerResolver
Transfer config.TransferPolicy
DistributorVersion string
@@ -48,9 +50,11 @@ type Plan struct {
BundleID string
BundlePath string
DestinationBundlePath string
PathMapping string
Action Action
Reason string
Force bool
PrimaryURL string
Outputs []Output
ExistingState *state.DistributorState
}
@@ -60,6 +64,7 @@ type Output struct {
DestinationPath string
Kind string
Transform string
URL string
Data []byte
SHA256 string
Size int64
@@ -73,11 +78,15 @@ func Build(ctx context.Context, req Request) (Plan, error) {
if err != nil {
return Plan{}, err
}
outputs, primaryURL, err := PlanLinks(req, outputs)
if err != nil {
return Plan{}, err
}
status, err := inspectDestination(ctx, req.DestinationBackend, req.DestinationBundlePath)
if err != nil {
return Plan{}, err
}
comparison := state.Compare(req.SourceBundle.Manifest, req.PipelineID, req.DestinationID, status)
comparison := compareDestination(req, status)
action, reason := actionForComparison(comparison, req.Transfer, req.Force)
plan := Plan{
PipelineID: req.PipelineID,
@@ -85,9 +94,11 @@ func Build(ctx context.Context, req Request) (Plan, error) {
BundleID: req.SourceBundle.Manifest.ID,
BundlePath: req.SourceBundle.RootRelativePath,
DestinationBundlePath: req.DestinationBundlePath,
PathMapping: req.PathMapping,
Action: action,
Reason: reason,
Force: action == ActionForceReplace,
PrimaryURL: primaryURL,
Outputs: outputs,
ExistingState: status.State,
}
@@ -116,6 +127,21 @@ func validateRequest(req Request) error {
return nil
}
func compareDestination(req Request, status state.DestinationStatus) state.Comparison {
comparison := state.Compare(req.SourceBundle.Manifest, req.PipelineID, req.DestinationID, status)
if req.PathMapping != config.PathMappingFixed || comparison.Outcome != state.OutcomeDifferentSourceConflict || status.State == nil {
return comparison
}
destinationManifest := status.State.Source.Manifest
if destinationManifest.Created.Before(req.SourceBundle.Manifest.Created) {
return state.Comparison{Outcome: state.OutcomeDestinationOlder, Reason: "fixed destination source is older than selected source"}
}
if destinationManifest.Created.After(req.SourceBundle.Manifest.Created) {
return state.Comparison{Outcome: state.OutcomeDestinationNewer, Reason: "fixed destination source is newer than selected source"}
}
return comparison
}
func actionForComparison(comparison state.Comparison, transfer config.TransferPolicy, force bool) (Action, string) {
switch comparison.Outcome {
case state.OutcomeDestinationAbsent:

View File

@@ -19,6 +19,7 @@ type DistributorState struct {
DestinationID string
PublishedAt time.Time
Source SourceState
Links *LinkState
Outputs []OutputFile
}
@@ -26,11 +27,16 @@ type SourceState struct {
Manifest bundle.Manifest
}
type LinkState struct {
PrimaryURL string
}
type OutputFile struct {
Path string
Kind string
SourcePath string
Transform string
URL string
SHA256 string
Size int64
}
@@ -42,6 +48,7 @@ type rawDistributorState struct {
DestinationID *string `json:"destination_id"`
PublishedAt *string `json:"published_at"`
Source *rawSourceState `json:"source"`
Links *rawLinkState `json:"links"`
Outputs []rawOutputFile `json:"outputs"`
}
@@ -49,11 +56,16 @@ type rawSourceState struct {
Manifest json.RawMessage `json:"manifest"`
}
type rawLinkState struct {
PrimaryURL string `json:"primary_url"`
}
type rawOutputFile struct {
Path *string `json:"path"`
Kind *string `json:"kind"`
SourcePath *string `json:"source_path"`
Transform string `json:"transform"`
URL string `json:"url"`
SHA256 *string `json:"sha256"`
Size *int64 `json:"size"`
}
@@ -112,6 +124,9 @@ func parseRaw(raw rawDistributorState) (DistributorState, error) {
return DistributorState{}, fmt.Errorf("state source.manifest: %w", err)
}
state.Source.Manifest = manifest
if raw.Links != nil {
state.Links = &LinkState{PrimaryURL: raw.Links.PrimaryURL}
}
if raw.Outputs == nil {
return DistributorState{}, fmt.Errorf("state outputs is required")
}
@@ -161,6 +176,7 @@ func parseOutput(index int, raw rawOutputFile) (OutputFile, error) {
Kind: *raw.Kind,
SourcePath: *raw.SourcePath,
Transform: raw.Transform,
URL: raw.URL,
SHA256: *raw.SHA256,
Size: *raw.Size,
}, nil
@@ -181,6 +197,7 @@ func (s DistributorState) MarshalJSON() ([]byte, error) {
DestinationID string `json:"destination_id"`
PublishedAt string `json:"published_at"`
Source sourceJSON `json:"source"`
Links *LinkState `json:"links,omitempty"`
Outputs []OutputFile `json:"outputs"`
}
return json.Marshal(stateJSON{
@@ -190,16 +207,25 @@ func (s DistributorState) MarshalJSON() ([]byte, error) {
DestinationID: s.DestinationID,
PublishedAt: s.PublishedAtString(),
Source: sourceJSON{Manifest: s.Source.Manifest},
Links: s.Links,
Outputs: s.Outputs,
})
}
func (l LinkState) MarshalJSON() ([]byte, error) {
type linkJSON struct {
PrimaryURL string `json:"primary_url,omitempty"`
}
return json.Marshal(linkJSON{PrimaryURL: l.PrimaryURL})
}
func (o OutputFile) MarshalJSON() ([]byte, error) {
type outputJSON struct {
Path string `json:"path"`
Kind string `json:"kind"`
SourcePath string `json:"source_path"`
Transform string `json:"transform,omitempty"`
URL string `json:"url,omitempty"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
@@ -208,6 +234,7 @@ func (o OutputFile) MarshalJSON() ([]byte, error) {
Kind: o.Kind,
SourcePath: o.SourcePath,
Transform: o.Transform,
URL: o.URL,
SHA256: o.SHA256,
Size: o.Size,
})

View File

@@ -29,6 +29,22 @@ func TestParseValidState(t *testing.T) {
}
}
func TestParseValidStateWithLinks(t *testing.T) {
body := strings.Replace(validStateJSON(t), `"outputs": [`, `"links": {"primary_url": "https://reports.example.com/archive/report.md"},`+"\n "+`"outputs": [`, 1)
body = strings.Replace(body, `"source_path": "report.md",`, `"source_path": "report.md",`+"\n "+`"url": "https://reports.example.com/archive/report.md",`, 1)
state, err := Parse([]byte(body))
if err != nil {
t.Fatalf("Parse() error = %v", err)
}
if state.Links == nil || state.Links.PrimaryURL != "https://reports.example.com/archive/report.md" {
t.Fatalf("links = %#v, want primary URL", state.Links)
}
if state.Outputs[0].URL != "https://reports.example.com/archive/report.md" {
t.Fatalf("output URL = %q", state.Outputs[0].URL)
}
}
func TestParseNormalizesPublishedAtOffset(t *testing.T) {
body := strings.Replace(validStateJSON(t), `"published_at": "2026-05-30T11:12:00Z"`, `"published_at": "2026-05-30T13:12:00+02:00"`, 1)
state, err := Parse([]byte(body))
@@ -145,6 +161,12 @@ func TestParseRejectsInvalidOutputMetadata(t *testing.T) {
"negative size": func(s *DistributorState) {
s.Outputs[0].Size = -1
},
"invalid output url": func(s *DistributorState) {
s.Outputs[0].URL = "file:///tmp/report.md"
},
"invalid primary url": func(s *DistributorState) {
s.Links = &LinkState{PrimaryURL: "file:///tmp/report.md"}
},
}
for name, mutate := range tests {
t.Run(name, func(t *testing.T) {
@@ -188,6 +210,36 @@ func TestMarshalNormalizesPublishedAtUTC(t *testing.T) {
}
}
func TestMarshalIncludesLinksWhenPresent(t *testing.T) {
source := validManifest(t)
state := DistributorState{
SchemaVersion: SchemaVersion,
PipelineID: "reports",
DestinationID: "archive",
PublishedAt: time.Date(2026, 5, 30, 11, 12, 0, 0, time.UTC),
Source: SourceState{Manifest: source},
Links: &LinkState{PrimaryURL: "https://reports.example.com/archive/report.md"},
Outputs: []OutputFile{{
Path: "report.md",
Kind: OutputKindSource,
SourcePath: "report.md",
URL: "https://reports.example.com/archive/report.md",
SHA256: source.Files[0].SHA256,
Size: source.Files[0].Size,
}},
}
data, err := json.Marshal(state)
if err != nil {
t.Fatalf("Marshal() error = %v", err)
}
if !strings.Contains(string(data), `"links":{"primary_url":"https://reports.example.com/archive/report.md"}`) {
t.Fatalf("json = %s, want links primary URL", data)
}
if !strings.Contains(string(data), `"url":"https://reports.example.com/archive/report.md"`) {
t.Fatalf("json = %s, want output URL", data)
}
}
func validStateJSON(t *testing.T) string {
t.Helper()
return validStateWithManifestJSON(t, manifestJSON(t))

View File

@@ -2,6 +2,7 @@ package state
import (
"fmt"
"net/url"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
@@ -28,6 +29,11 @@ func Validate(s DistributorState) error {
if err := validateEmbeddedManifest(s.Source.Manifest); err != nil {
return fmt.Errorf("state source.manifest: %w", err)
}
if s.Links != nil && s.Links.PrimaryURL != "" {
if err := validateStateURL(s.Links.PrimaryURL); err != nil {
return fmt.Errorf("state links.primary_url: %w", err)
}
}
if s.Outputs == nil {
return fmt.Errorf("state outputs is required")
}
@@ -63,6 +69,11 @@ func validateOutput(index int, output OutputFile) error {
if output.Kind == OutputKindGenerated && output.Transform == "" {
return fmt.Errorf("state outputs[%d].transform is required for generated output", index)
}
if output.URL != "" {
if err := validateStateURL(output.URL); err != nil {
return fmt.Errorf("state outputs[%d].url: %w", index, err)
}
}
if err := bundle.ValidateDigest(output.SHA256); err != nil {
return fmt.Errorf("state outputs[%d].sha256: %w", index, err)
}
@@ -71,3 +82,23 @@ func validateOutput(index int, output OutputFile) error {
}
return nil
}
func validateStateURL(value string) error {
parsed, err := url.Parse(value)
if err != nil {
return err
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("must use http or https")
}
if parsed.Host == "" {
return fmt.Errorf("must include a host")
}
if parsed.RawQuery != "" {
return fmt.Errorf("must not include a query string")
}
if parsed.Fragment != "" {
return fmt.Errorf("must not include a fragment")
}
return nil
}

View File

@@ -25,24 +25,26 @@ func (t *Transformer) Generate(ctx context.Context, req transform.Request) ([]tr
if t.renderer == nil {
t.renderer = goldmark.New()
}
switch markdownMode(req.Markdown.Mode) {
case transform.MarkdownModeSidecar:
return t.generateSidecars(ctx, req)
case transform.MarkdownModeIndex:
return t.generateIndex(ctx, req)
default:
return nil, fmt.Errorf("markdown mode %q is not supported", req.Markdown.Mode)
}
}
func (t *Transformer) generateSidecars(ctx context.Context, req transform.Request) ([]transform.Output, error) {
var outputs []transform.Output
for _, file := range req.SourceBundle.Manifest.Files {
if !strings.HasSuffix(file.Path, ".md") {
continue
}
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, file.Path)
html, err := t.render(ctx, req, file.Path)
if err != nil {
return nil, err
}
data, err := req.SourceBackend.ReadFile(ctx, sourcePath)
if err != nil {
return nil, fmt.Errorf("read markdown source %q: %w", file.Path, err)
}
var rendered bytes.Buffer
if err := t.renderer.Convert(data, &rendered); err != nil {
return nil, fmt.Errorf("render markdown source %q: %w", file.Path, err)
}
html := wrapHTML(rendered.Bytes())
outputPath := strings.TrimSuffix(file.Path, ".md") + ".html"
outputs = append(outputs, transform.Output{
Path: outputPath,
@@ -55,3 +57,78 @@ func (t *Transformer) Generate(ctx context.Context, req transform.Request) ([]tr
}
return outputs, nil
}
func (t *Transformer) generateIndex(ctx context.Context, req transform.Request) ([]transform.Output, error) {
input, err := selectIndexInput(req)
if err != nil {
return nil, err
}
html, err := t.render(ctx, req, input)
if err != nil {
return nil, err
}
return []transform.Output{{
Path: "index.html",
SourcePath: input,
Transform: transform.MarkdownToHTML,
Data: html,
SHA256: bundle.FileDigest(html),
Size: int64(len(html)),
}}, nil
}
func selectIndexInput(req transform.Request) (string, error) {
if req.Markdown.Input != "" {
if err := storage.ValidatePath(req.Markdown.Input); err != nil {
return "", fmt.Errorf("markdown input %q: %w", req.Markdown.Input, err)
}
for _, file := range req.SourceBundle.Manifest.Files {
if file.Path != req.Markdown.Input {
continue
}
if !strings.HasSuffix(file.Path, ".md") {
return "", fmt.Errorf("markdown input %q must end in .md", req.Markdown.Input)
}
return file.Path, nil
}
return "", fmt.Errorf("markdown input %q is not listed in the source manifest", req.Markdown.Input)
}
var markdownFiles []string
for _, file := range req.SourceBundle.Manifest.Files {
if strings.HasSuffix(file.Path, ".md") {
markdownFiles = append(markdownFiles, file.Path)
}
}
switch len(markdownFiles) {
case 0:
return "", fmt.Errorf("markdown index mode requires one markdown source file or transform.markdown_to_html.input")
case 1:
return markdownFiles[0], nil
default:
return "", fmt.Errorf("markdown index mode found multiple markdown source files; set transform.markdown_to_html.input")
}
}
func (t *Transformer) render(ctx context.Context, req transform.Request, sourceFile string) ([]byte, error) {
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, sourceFile)
if err != nil {
return nil, err
}
data, err := req.SourceBackend.ReadFile(ctx, sourcePath)
if err != nil {
return nil, fmt.Errorf("read markdown source %q: %w", sourceFile, err)
}
var rendered bytes.Buffer
if err := t.renderer.Convert(data, &rendered); err != nil {
return nil, fmt.Errorf("render markdown source %q: %w", sourceFile, err)
}
return wrapHTML(rendered.Bytes()), nil
}
func markdownMode(mode string) string {
if mode == "" {
return transform.MarkdownModeSidecar
}
return mode
}

View File

@@ -40,6 +40,124 @@ func TestGenerateMarkdownSidecar(t *testing.T) {
}
}
func TestGenerateMarkdownIndexExplicitInput(t *testing.T) {
backend := fake.New()
sourceBundle := testutil.WriteFakeSourceBundle(t, backend, "", testutil.BundleOptions{
ID: "bundle",
Files: []testutil.SourceFile{
{Path: "report.md", Data: "# Report\n"},
{Path: "notes.md", Data: "# Notes\n"},
},
})
outputs, err := New().Generate(context.Background(), transform.Request{
SourceBackend: backend,
SourceBundle: sourceBundle,
Markdown: transform.MarkdownOptions{Mode: transform.MarkdownModeIndex, Input: "notes.md"},
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
if got, want := len(outputs), 1; got != want {
t.Fatalf("output count = %d, want %d", got, want)
}
output := outputs[0]
if output.Path != "index.html" || output.SourcePath != "notes.md" || output.Transform != transform.MarkdownToHTML {
t.Fatalf("output metadata = %#v, want index from notes.md", output)
}
if !strings.Contains(string(output.Data), "<h1>Notes</h1>") {
t.Fatalf("html = %q, want notes content", output.Data)
}
if output.SHA256 != bundle.FileDigest(output.Data) || output.Size != int64(len(output.Data)) {
t.Fatalf("digest/size metadata = %s/%d", output.SHA256, output.Size)
}
}
func TestGenerateMarkdownIndexSelectsOnlyMarkdownFile(t *testing.T) {
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
outputs, err := New().Generate(context.Background(), transform.Request{
SourceBackend: backend,
SourceBundle: sourceBundle,
Markdown: transform.MarkdownOptions{Mode: transform.MarkdownModeIndex},
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
if got, want := outputs[0].Path, "index.html"; got != want {
t.Fatalf("path = %q, want %q", got, want)
}
if got, want := outputs[0].SourcePath, "report.md"; got != want {
t.Fatalf("source path = %q, want %q", got, want)
}
}
func TestGenerateMarkdownIndexRejectsAmbiguousInput(t *testing.T) {
backend := fake.New()
sourceBundle := testutil.WriteFakeSourceBundle(t, backend, "", testutil.BundleOptions{
ID: "bundle",
Files: []testutil.SourceFile{
{Path: "report.md", Data: "# Report\n"},
{Path: "notes.md", Data: "# Notes\n"},
},
})
_, err := New().Generate(context.Background(), transform.Request{
SourceBackend: backend,
SourceBundle: sourceBundle,
Markdown: transform.MarkdownOptions{Mode: transform.MarkdownModeIndex},
})
if err == nil || !strings.Contains(err.Error(), "multiple markdown source files") {
t.Fatalf("Generate() error = %v, want ambiguous input error", err)
}
}
func TestGenerateMarkdownIndexRejectsMissingMarkdown(t *testing.T) {
backend := fake.New()
sourceBundle := testutil.WriteFakeSourceBundle(t, backend, "", testutil.BundleOptions{
ID: "bundle",
Files: []testutil.SourceFile{{Path: "summary.txt", Data: "Summary\n"}},
})
_, err := New().Generate(context.Background(), transform.Request{
SourceBackend: backend,
SourceBundle: sourceBundle,
Markdown: transform.MarkdownOptions{Mode: transform.MarkdownModeIndex},
})
if err == nil || !strings.Contains(err.Error(), "requires one markdown source file") {
t.Fatalf("Generate() error = %v, want missing markdown error", err)
}
}
func TestGenerateMarkdownIndexRejectsInvalidExplicitInput(t *testing.T) {
backend, sourceBundle := markdownFixture(t, "# Title\n")
tests := []struct {
name string
input string
wantError string
}{
{name: "unsafe", input: "../report.md", wantError: "markdown input"},
{name: "not listed", input: "missing.md", wantError: "not listed"},
{name: "not markdown", input: "summary.txt", wantError: "must end in .md"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := New().Generate(context.Background(), transform.Request{
SourceBackend: backend,
SourceBundle: sourceBundle,
Markdown: transform.MarkdownOptions{Mode: transform.MarkdownModeIndex, Input: tt.input},
})
if err == nil || !strings.Contains(err.Error(), tt.wantError) {
t.Fatalf("Generate() error = %v, want substring %q", err, tt.wantError)
}
})
}
}
func TestGenerateIgnoresNonMarkdown(t *testing.T) {
backend := fake.New()
if _, err := backend.WriteFile(context.Background(), "summary.txt", []byte("Summary\n"), storage.WriteOptions{}); err != nil {

View File

@@ -1,3 +1,8 @@
package transform
const MarkdownToHTML = "markdown_to_html"
const (
MarkdownModeSidecar = "sidecar"
MarkdownModeIndex = "index"
)

View File

@@ -19,6 +19,12 @@ type Output struct {
type Request struct {
SourceBundle bundle.Bundle
SourceBackend storage.Backend
Markdown MarkdownOptions
}
type MarkdownOptions struct {
Mode string
Input string
}
type Transformer interface {

118
pkg/bundle/build.go Normal file
View File

@@ -0,0 +1,118 @@
package bundle
import (
"fmt"
"os"
"path"
"path/filepath"
"sort"
"time"
)
func BuildManifest(opts BuildOptions) (Manifest, error) {
if opts.Root == "" {
return Manifest{}, fmt.Errorf("root is required")
}
if opts.ID == "" {
return Manifest{}, fmt.Errorf("id is required")
}
explicit := len(opts.Files) > 0
if explicit == opts.Scan {
return Manifest{}, fmt.Errorf("select exactly one file mode")
}
created := opts.Created
if created.IsZero() {
created = time.Now().UTC()
}
paths := append([]string(nil), opts.Files...)
var err error
if opts.Scan {
paths, err = scanSourcePaths(opts.Root)
if err != nil {
return Manifest{}, err
}
}
files := make([]ManifestFile, 0, len(paths))
for _, sourcePath := range paths {
file, err := buildManifestFile(opts.Root, sourcePath)
if err != nil {
return Manifest{}, err
}
files = append(files, file)
}
manifest := Manifest{
SchemaVersion: SchemaVersion,
ID: opts.ID,
Created: created,
Files: files,
}
manifest.Digest = BundleDigest(manifest.Files)
if err := ValidateManifest(manifest); err != nil {
return Manifest{}, err
}
return manifest, nil
}
func scanSourcePaths(root string) ([]string, error) {
var paths []string
err := filepath.WalkDir(root, func(filePath string, entry os.DirEntry, err error) error {
if err != nil {
return err
}
if filePath == root {
return nil
}
relative, err := filepath.Rel(root, filePath)
if err != nil {
return err
}
sourcePath := filepath.ToSlash(relative)
if entry.Type()&os.ModeSymlink != 0 {
return fmt.Errorf("source file %q must be a regular file", sourcePath)
}
if entry.IsDir() {
return nil
}
if path.Base(sourcePath) == ManifestName || path.Base(sourcePath) == distributorStateName {
return nil
}
info, err := entry.Info()
if err != nil {
return err
}
if !info.Mode().IsRegular() {
return fmt.Errorf("source file %q must be a regular file", sourcePath)
}
paths = append(paths, sourcePath)
return nil
})
if err != nil {
return nil, err
}
sort.Strings(paths)
return paths, nil
}
func buildManifestFile(root, sourcePath string) (ManifestFile, error) {
if err := ValidateSourcePath(sourcePath); err != nil {
return ManifestFile{}, fmt.Errorf("source file %q: %w", sourcePath, err)
}
fullPath := filepath.Join(root, filepath.FromSlash(sourcePath))
info, err := os.Lstat(fullPath)
if err != nil {
return ManifestFile{}, fmt.Errorf("source file %q stat: %w", sourcePath, err)
}
if !info.Mode().IsRegular() {
return ManifestFile{}, fmt.Errorf("source file %q must be a regular file", sourcePath)
}
data, err := os.ReadFile(fullPath)
if err != nil {
return ManifestFile{}, fmt.Errorf("source file %q read: %w", sourcePath, err)
}
return ManifestFile{
Path: sourcePath,
SHA256: FileDigest(data),
Size: int64(len(data)),
}, nil
}

363
pkg/bundle/bundle_test.go Normal file
View File

@@ -0,0 +1,363 @@
package bundle
import (
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
func TestBuildManifestExplicitFilesPreservesOrder(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "b.txt", "bravo")
writeFile(t, root, "a.txt", "alpha")
before := time.Now().UTC()
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.example",
Files: []string{"b.txt", "a.txt"},
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
after := time.Now().UTC()
paths := manifestPaths(manifest)
if want := []string{"b.txt", "a.txt"}; !reflect.DeepEqual(paths, want) {
t.Fatalf("paths = %v, want %v", paths, want)
}
if manifest.SchemaVersion != SchemaVersion {
t.Fatalf("schema version = %d, want %d", manifest.SchemaVersion, SchemaVersion)
}
if manifest.ID != "reports.example" {
t.Fatalf("id = %q", manifest.ID)
}
if manifest.Created.Before(before) || manifest.Created.After(after) {
t.Fatalf("created = %s, want between %s and %s", manifest.Created, before, after)
}
if err := ValidateManifest(manifest); err != nil {
t.Fatalf("ValidateManifest() error = %v", err)
}
if err := ValidateBundle(root, manifest); err != nil {
t.Fatalf("ValidateBundle() error = %v", err)
}
}
func TestBuildManifestScanSortsAndFiltersMetadata(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "z.txt", "zulu")
writeFile(t, root, "nested/.hidden", "hidden")
writeFile(t, root, ManifestName, "old manifest")
writeFile(t, root, distributorStateName, "state")
writeFile(t, root, "nested/manifest.json", "nested manifest")
writeFile(t, root, "nested/.distributor.json", "nested state")
created := time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC)
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.scan",
Created: created,
Scan: true,
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
if !manifest.Created.Equal(created) {
t.Fatalf("created = %s, want %s", manifest.Created, created)
}
paths := manifestPaths(manifest)
if want := []string{"nested/.hidden", "z.txt"}; !reflect.DeepEqual(paths, want) {
t.Fatalf("paths = %v, want %v", paths, want)
}
}
func TestBuildManifestRequiresOneFileMode(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
tests := []BuildOptions{
{Root: root, ID: "reports.none"},
{Root: root, ID: "reports.both", Files: []string{"report.txt"}, Scan: true},
}
for _, opts := range tests {
if _, err := BuildManifest(opts); err == nil {
t.Fatalf("BuildManifest(%+v) error = nil, want error", opts)
}
}
}
func TestBuildManifestRejectsUnsafePath(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
_, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.unsafe",
Files: []string{"../report.txt"},
})
if err == nil {
t.Fatal("BuildManifest() error = nil, want unsafe path error")
}
}
func TestBuildManifestRejectsSymlink(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "target.txt", "target")
if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
_, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.symlink",
Files: []string{"link.txt"},
})
if err == nil || !strings.Contains(err.Error(), "regular file") {
t.Fatalf("BuildManifest() error = %v, want regular file error", err)
}
}
func TestScanRejectsSymlink(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "target.txt", "target")
if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
_, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.scan.symlink",
Scan: true,
})
if err == nil || !strings.Contains(err.Error(), "regular file") {
t.Fatalf("BuildManifest() error = %v, want regular file error", err)
}
}
func TestDigestFunctionsUseCanonicalFilePayload(t *testing.T) {
files := []ManifestFile{
{Path: "report.md", SHA256: FileDigest([]byte("report")), Size: 6},
{Path: "summary.txt", SHA256: FileDigest([]byte("summary")), Size: 7},
}
payload := CanonicalFilePayload(files)
if want := `[{"path":"report.md","sha256":"sha256:845e91831319e89c4d656bdb80c278ac09a7230d61e5dfd2e1b1fbb436ac8917","size":6},{"path":"summary.txt","sha256":"sha256:761b7ad8ad439b2855fcbb611331c646ef0870b0631247bba3f3025cb6df5a53","size":7}]`; payload != want {
t.Fatalf("payload = %q, want %q", payload, want)
}
if digest := BundleDigest(files); !strings.HasPrefix(digest, "sha256:") || len(digest) != len("sha256:")+64 {
t.Fatalf("BundleDigest() = %q, want sha256 digest", digest)
}
}
func TestParseMarshalLoadAndWriteManifest(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.json",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []string{"report.txt"},
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
data, err := MarshalManifest(manifest)
if err != nil {
t.Fatalf("MarshalManifest() error = %v", err)
}
if !strings.HasSuffix(string(data), "\n") {
t.Fatalf("manifest JSON = %q, want trailing newline", string(data))
}
if !strings.Contains(string(data), `"schema_version": 1`) || !strings.Contains(string(data), `"sha256": "`) {
t.Fatalf("manifest JSON = %q, want fixed manifest fields", string(data))
}
parsed, err := ParseManifest(data)
if err != nil {
t.Fatalf("ParseManifest() error = %v", err)
}
if !reflect.DeepEqual(parsed, manifest) {
t.Fatalf("parsed manifest = %#v, want %#v", parsed, manifest)
}
if err := WriteManifest(root, manifest, WriteManifestOptions{}); err != nil {
t.Fatalf("WriteManifest() error = %v", err)
}
loaded, err := LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if !reflect.DeepEqual(loaded, manifest) {
t.Fatalf("loaded manifest = %#v, want %#v", loaded, manifest)
}
if err := WriteManifest(root, manifest, WriteManifestOptions{}); err == nil {
t.Fatal("WriteManifest() error = nil, want exists error")
}
manifest.ID = "reports.updated"
manifest.Digest = BundleDigest(manifest.Files)
if err := WriteManifest(root, manifest, WriteManifestOptions{Overwrite: true}); err != nil {
t.Fatalf("WriteManifest(overwrite) error = %v", err)
}
loaded, err = LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() after overwrite error = %v", err)
}
if loaded.ID != "reports.updated" {
t.Fatalf("loaded id = %q, want updated", loaded.ID)
}
}
func TestValidateManifestRejectsInvalidManifest(t *testing.T) {
created := time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC)
fileDigest := FileDigest([]byte("report"))
valid := Manifest{
SchemaVersion: SchemaVersion,
ID: "reports.valid",
Created: created,
Files: []ManifestFile{{Path: "report.txt", SHA256: fileDigest, Size: 6}},
}
valid.Digest = BundleDigest(valid.Files)
tests := map[string]func(Manifest) Manifest{
"schema version": func(manifest Manifest) Manifest {
manifest.SchemaVersion = 2
return manifest
},
"id": func(manifest Manifest) Manifest {
manifest.ID = ""
return manifest
},
"created": func(manifest Manifest) Manifest {
manifest.Created = time.Time{}
return manifest
},
"files": func(manifest Manifest) Manifest {
manifest.Files = nil
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"path": func(manifest Manifest) Manifest {
manifest.Files[0].Path = "manifest.json"
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"duplicate": func(manifest Manifest) Manifest {
manifest.Files = append(manifest.Files, manifest.Files[0])
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"size": func(manifest Manifest) Manifest {
manifest.Files[0].Size = -1
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"digest": func(manifest Manifest) Manifest {
manifest.Digest = "sha256:0000000000000000000000000000000000000000000000000000000000000000"
return manifest
},
}
for name, mutate := range tests {
t.Run(name, func(t *testing.T) {
if err := ValidateManifest(mutate(valid)); err == nil {
t.Fatal("ValidateManifest() error = nil, want error")
}
})
}
}
func TestValidateBundleRejectsLocalFileProblems(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.bundle",
Files: []string{"report.txt"},
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
if err := ValidateBundle(root, manifest); err != nil {
t.Fatalf("ValidateBundle() error = %v", err)
}
missing := cloneManifest(manifest)
missing.Files[0].Path = "missing.txt"
missing.Digest = BundleDigest(missing.Files)
if err := ValidateBundle(root, missing); err == nil {
t.Fatal("ValidateBundle() missing error = nil, want error")
}
sizeMismatch := cloneManifest(manifest)
sizeMismatch.Files[0].Size++
sizeMismatch.Digest = BundleDigest(sizeMismatch.Files)
if err := ValidateBundle(root, sizeMismatch); err == nil || !strings.Contains(err.Error(), "size mismatch") {
t.Fatalf("ValidateBundle() size error = %v, want size mismatch", err)
}
digestMismatch := cloneManifest(manifest)
writeFile(t, root, "report.txt", "change")
if err := ValidateBundle(root, digestMismatch); err == nil || !strings.Contains(err.Error(), "sha256 mismatch") {
t.Fatalf("ValidateBundle() digest error = %v, want digest mismatch", err)
}
}
func TestValidateBundleRejectsSymlink(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "target.txt", "target")
if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
file := ManifestFile{Path: "link.txt", SHA256: FileDigest([]byte("target")), Size: 6}
manifest := Manifest{
SchemaVersion: SchemaVersion,
ID: "reports.link",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []ManifestFile{file},
}
manifest.Digest = BundleDigest(manifest.Files)
err := ValidateBundle(root, manifest)
if err == nil || !strings.Contains(err.Error(), "regular file") {
t.Fatalf("ValidateBundle() error = %v, want regular file error", err)
}
}
func TestValidateSourcePath(t *testing.T) {
valid := []string{"report.md", "nested/report.md", ".well-known/report.txt"}
for _, path := range valid {
if err := ValidateSourcePath(path); err != nil {
t.Fatalf("ValidateSourcePath(%q) error = %v", path, err)
}
}
invalid := []string{"", "../report.md", "/report.md", "nested/../report.md", `nested\report.md`, ManifestName, distributorStateName}
for _, path := range invalid {
if err := ValidateSourcePath(path); err == nil {
t.Fatalf("ValidateSourcePath(%q) error = nil, want error", path)
}
}
}
func manifestPaths(manifest Manifest) []string {
paths := make([]string, 0, len(manifest.Files))
for _, file := range manifest.Files {
paths = append(paths, file.Path)
}
return paths
}
func cloneManifest(manifest Manifest) Manifest {
manifest.Files = append([]ManifestFile(nil), manifest.Files...)
return manifest
}
func writeFile(t *testing.T, root, relative, body string) {
t.Helper()
path := filepath.Join(root, filepath.FromSlash(relative))
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("mkdir %s: %v", filepath.Dir(path), err)
}
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatalf("write %s: %v", path, err)
}
}

49
pkg/bundle/digest.go Normal file
View File

@@ -0,0 +1,49 @@
package bundle
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"regexp"
"strconv"
"strings"
)
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
func validateDigest(value string) error {
if !digestPattern.MatchString(value) {
return fmt.Errorf("must be lowercase sha256:<64 hex>")
}
return nil
}
func FileDigest(data []byte) string {
sum := sha256.Sum256(data)
return "sha256:" + hex.EncodeToString(sum[:])
}
func BundleDigest(files []ManifestFile) string {
canonical := CanonicalFilePayload(files)
sum := sha256.Sum256([]byte(canonical))
return "sha256:" + hex.EncodeToString(sum[:])
}
func CanonicalFilePayload(files []ManifestFile) string {
var builder strings.Builder
builder.WriteByte('[')
for index, file := range files {
if index > 0 {
builder.WriteByte(',')
}
builder.WriteString(`{"path":`)
builder.WriteString(strconv.Quote(file.Path))
builder.WriteString(`,"sha256":`)
builder.WriteString(strconv.Quote(file.SHA256))
builder.WriteString(`,"size":`)
builder.WriteString(strconv.FormatInt(file.Size, 10))
builder.WriteByte('}')
}
builder.WriteByte(']')
return builder.String()
}

9
pkg/bundle/doc.go Normal file
View File

@@ -0,0 +1,9 @@
// Package bundle provides producer-facing helpers for distributor source
// bundle manifests.
//
// A source bundle is a local directory containing a manifest.json file and the
// files listed by that manifest. This package owns the public manifest model,
// digest calculation, path validation, manifest parsing, manifest building,
// local bundle writing, and local bundle validation used by Go producer
// applications.
package bundle

View File

@@ -0,0 +1,73 @@
package bundle_test
import (
"fmt"
"os"
"path/filepath"
"time"
"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
)
func ExampleBuildManifest() {
root, err := os.MkdirTemp("", "distributor-bundle-*")
if err != nil {
panic(err)
}
defer os.RemoveAll(root)
if err := os.WriteFile(filepath.Join(root, "report.txt"), []byte("report\n"), 0o600); err != nil {
panic(err)
}
manifest, err := bundle.BuildManifest(bundle.BuildOptions{
Root: root,
ID: "reports.example",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []string{"report.txt"},
})
if err != nil {
panic(err)
}
fmt.Println(manifest.ID)
fmt.Println(manifest.Files[0].Path)
fmt.Println(manifest.Files[0].Size)
// Output:
// reports.example
// report.txt
// 7
}
func ExampleWriteBundle() {
sourceRoot, err := os.MkdirTemp("", "distributor-source-*")
if err != nil {
panic(err)
}
defer os.RemoveAll(sourceRoot)
outputRoot := filepath.Join(os.TempDir(), "distributor-bundle-example")
defer os.RemoveAll(outputRoot)
if err := os.WriteFile(filepath.Join(sourceRoot, "report.txt"), []byte("report\n"), 0o600); err != nil {
panic(err)
}
manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
Root: outputRoot,
ID: "reports.example",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []bundle.BundleFile{
{SourcePath: filepath.Join(sourceRoot, "report.txt"), Path: "report.txt"},
},
Overwrite: true,
})
if err != nil {
panic(err)
}
fmt.Println(manifest.ID)
fmt.Println(manifest.Files[0].Path)
// Output:
// reports.example
// report.txt
}

160
pkg/bundle/manifest.go Normal file
View File

@@ -0,0 +1,160 @@
package bundle
import (
"bytes"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"time"
)
type rawManifest struct {
SchemaVersion *int `json:"schema_version"`
ID *string `json:"id"`
Digest *string `json:"digest"`
Created *string `json:"created"`
Files []rawManifestFile `json:"files"`
}
type rawManifestFile struct {
Path *string `json:"path"`
SHA256 *string `json:"sha256"`
Size *int64 `json:"size"`
}
func ParseManifest(data []byte) (Manifest, error) {
decoder := json.NewDecoder(bytes.NewReader(data))
var raw rawManifest
if err := decoder.Decode(&raw); err != nil {
return Manifest{}, fmt.Errorf("parse manifest: %w", err)
}
var extra any
if err := decoder.Decode(&extra); err != io.EOF {
return Manifest{}, fmt.Errorf("parse manifest: trailing data")
}
var manifest Manifest
if raw.SchemaVersion == nil {
return Manifest{}, fmt.Errorf("manifest schema_version is required")
}
manifest.SchemaVersion = *raw.SchemaVersion
if raw.ID == nil || *raw.ID == "" {
return Manifest{}, fmt.Errorf("manifest id is required")
}
manifest.ID = *raw.ID
if raw.Digest == nil || *raw.Digest == "" {
return Manifest{}, fmt.Errorf("manifest digest is required")
}
manifest.Digest = *raw.Digest
if raw.Created == nil || *raw.Created == "" {
return Manifest{}, fmt.Errorf("manifest created is required")
}
created, err := time.Parse(time.RFC3339, *raw.Created)
if err != nil {
return Manifest{}, fmt.Errorf("manifest created must be RFC3339: %w", err)
}
manifest.Created = created
if len(raw.Files) == 0 {
return Manifest{}, fmt.Errorf("manifest files is required")
}
for index, rawFile := range raw.Files {
file, err := parseManifestFile(index, rawFile)
if err != nil {
return Manifest{}, err
}
manifest.Files = append(manifest.Files, file)
}
if err := ValidateManifest(manifest); err != nil {
return Manifest{}, fmt.Errorf("manifest %w", err)
}
return manifest, nil
}
func parseManifestFile(index int, raw rawManifestFile) (ManifestFile, error) {
if raw.Path == nil || *raw.Path == "" {
return ManifestFile{}, fmt.Errorf("manifest files[%d].path is required", index)
}
if raw.SHA256 == nil || *raw.SHA256 == "" {
return ManifestFile{}, fmt.Errorf("manifest files[%d].sha256 is required", index)
}
if raw.Size == nil {
return ManifestFile{}, fmt.Errorf("manifest files[%d].size is required", index)
}
return ManifestFile{
Path: *raw.Path,
SHA256: *raw.SHA256,
Size: *raw.Size,
}, nil
}
func MarshalManifest(manifest Manifest) ([]byte, error) {
if err := ValidateManifest(manifest); err != nil {
return nil, err
}
data, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
}
func LoadManifest(root string) (Manifest, error) {
data, err := os.ReadFile(filepath.Join(root, ManifestName))
if err != nil {
return Manifest{}, fmt.Errorf("read manifest: %w", err)
}
return ParseManifest(data)
}
func WriteManifest(root string, manifest Manifest, opts WriteManifestOptions) error {
data, err := MarshalManifest(manifest)
if err != nil {
return err
}
manifestPath := filepath.Join(root, ManifestName)
if !opts.Overwrite {
file, err := os.OpenFile(manifestPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o666)
if err != nil {
return fmt.Errorf("write manifest: %w", err)
}
if _, err := file.Write(data); err != nil {
_ = file.Close()
return fmt.Errorf("write manifest: %w", err)
}
if err := file.Close(); err != nil {
return fmt.Errorf("write manifest: %w", err)
}
return nil
}
dir := root
if dir == "" {
dir = "."
}
tmp, err := os.CreateTemp(dir, ".manifest-*.tmp")
if err != nil {
return fmt.Errorf("write manifest temp: %w", err)
}
tmpPath := tmp.Name()
cleanup := true
defer func() {
if cleanup {
_ = os.Remove(tmpPath)
}
}()
if _, err := tmp.Write(data); err != nil {
_ = tmp.Close()
return fmt.Errorf("write manifest temp: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("write manifest temp: %w", err)
}
if err := os.Rename(tmpPath, manifestPath); err != nil {
return fmt.Errorf("replace manifest: %w", err)
}
cleanup = false
return nil
}

31
pkg/bundle/path.go Normal file
View File

@@ -0,0 +1,31 @@
package bundle
import (
"fmt"
"path"
"strings"
)
const distributorStateName = ".distributor.json"
func ValidateSourcePath(value string) error {
if value == "" {
return fmt.Errorf("source path is required")
}
if strings.Contains(value, "\\") || strings.HasPrefix(value, "/") {
return fmt.Errorf("source path %q must be a clean relative slash-separated path", value)
}
if path.Clean(value) != value {
return fmt.Errorf("source path %q must be a clean relative slash-separated path", value)
}
for _, segment := range strings.Split(value, "/") {
if segment == "" || segment == "." || segment == ".." {
return fmt.Errorf("source path %q must be a clean relative slash-separated path", value)
}
}
switch value {
case ManifestName, distributorStateName:
return fmt.Errorf("%q is reserved", value)
}
return nil
}

46
pkg/bundle/types.go Normal file
View File

@@ -0,0 +1,46 @@
package bundle
import "time"
const ManifestName = "manifest.json"
const SchemaVersion = 1
type Manifest struct {
SchemaVersion int `json:"schema_version"`
ID string `json:"id"`
Digest string `json:"digest"`
Created time.Time `json:"created"`
Files []ManifestFile `json:"files"`
}
type ManifestFile struct {
Path string `json:"path"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
type BuildOptions struct {
Root string
ID string
Created time.Time
Files []string
Scan bool
}
type WriteManifestOptions struct {
Overwrite bool
}
type BundleFile struct {
SourcePath string
Path string
}
type WriteBundleOptions struct {
Root string
ID string
Created time.Time
Files []BundleFile
Overwrite bool
}

79
pkg/bundle/validate.go Normal file
View File

@@ -0,0 +1,79 @@
package bundle
import (
"fmt"
"os"
"path/filepath"
)
func ValidateManifest(manifest Manifest) error {
if manifest.SchemaVersion != SchemaVersion {
return fmt.Errorf("schema_version must be %d", SchemaVersion)
}
if manifest.ID == "" {
return fmt.Errorf("id is required")
}
if err := validateDigest(manifest.Digest); err != nil {
return fmt.Errorf("digest: %w", err)
}
if manifest.Created.IsZero() {
return fmt.Errorf("created is required")
}
if len(manifest.Files) == 0 {
return fmt.Errorf("files is required")
}
seen := make(map[string]struct{}, len(manifest.Files))
for index, file := range manifest.Files {
if err := ValidateSourcePath(file.Path); err != nil {
return fmt.Errorf("files[%d].path: %w", index, err)
}
if err := validateDigest(file.SHA256); err != nil {
return fmt.Errorf("files[%d].sha256: %w", index, err)
}
if file.Size < 0 {
return fmt.Errorf("files[%d].size must be non-negative", index)
}
if _, exists := seen[file.Path]; exists {
return fmt.Errorf("files[%d].path duplicates %q", index, file.Path)
}
seen[file.Path] = struct{}{}
}
if actual := BundleDigest(manifest.Files); actual != manifest.Digest {
return fmt.Errorf("digest mismatch: got %s want %s", actual, manifest.Digest)
}
return nil
}
func ValidateBundle(root string, manifest Manifest) error {
if err := ValidateManifest(manifest); err != nil {
return err
}
files := append([]ManifestFile(nil), manifest.Files...)
for index, manifestFile := range files {
fullPath := filepath.Join(root, filepath.FromSlash(manifestFile.Path))
info, err := os.Lstat(fullPath)
if err != nil {
return fmt.Errorf("file %q stat: %w", manifestFile.Path, err)
}
if !info.Mode().IsRegular() {
return fmt.Errorf("file %q must be a regular file", manifestFile.Path)
}
if info.Size() != manifestFile.Size {
return fmt.Errorf("file %q size mismatch: got %d want %d", manifestFile.Path, info.Size(), manifestFile.Size)
}
data, err := os.ReadFile(fullPath)
if err != nil {
return fmt.Errorf("file %q read: %w", manifestFile.Path, err)
}
actualDigest := FileDigest(data)
if actualDigest != manifestFile.SHA256 {
return fmt.Errorf("file %q sha256 mismatch: got %s want %s", manifestFile.Path, actualDigest, manifestFile.SHA256)
}
files[index].SHA256 = actualDigest
files[index].Size = int64(len(data))
}
if actualDigest := BundleDigest(files); actualDigest != manifest.Digest {
return fmt.Errorf("digest mismatch: got %s want %s", actualDigest, manifest.Digest)
}
return nil
}

187
pkg/bundle/writer.go Normal file
View File

@@ -0,0 +1,187 @@
package bundle
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
)
func WriteBundle(opts WriteBundleOptions) (Manifest, error) {
root, err := cleanBundleRoot(opts.Root)
if err != nil {
return Manifest{}, err
}
if opts.ID == "" {
return Manifest{}, fmt.Errorf("id is required")
}
if len(opts.Files) == 0 {
return Manifest{}, fmt.Errorf("files is required")
}
parent := filepath.Dir(root)
if err := os.MkdirAll(parent, 0o755); err != nil {
return Manifest{}, fmt.Errorf("create bundle parent: %w", err)
}
if !opts.Overwrite {
if _, err := os.Lstat(root); err == nil {
return Manifest{}, fmt.Errorf("bundle root %q already exists", root)
} else if !errors.Is(err, os.ErrNotExist) {
return Manifest{}, fmt.Errorf("stat bundle root: %w", err)
}
}
tempRoot, err := os.MkdirTemp(parent, "."+filepath.Base(root)+"-*.tmp")
if err != nil {
return Manifest{}, fmt.Errorf("create bundle temp root: %w", err)
}
removeTemp := true
defer func() {
if removeTemp {
_ = os.RemoveAll(tempRoot)
}
}()
paths, err := copyBundleFiles(tempRoot, opts.Files)
if err != nil {
return Manifest{}, err
}
manifest, err := BuildManifest(BuildOptions{
Root: tempRoot,
ID: opts.ID,
Created: opts.Created,
Files: paths,
})
if err != nil {
return Manifest{}, err
}
if err := WriteManifest(tempRoot, manifest, WriteManifestOptions{}); err != nil {
return Manifest{}, err
}
if err := ValidateBundle(tempRoot, manifest); err != nil {
return Manifest{}, err
}
if err := promoteBundleRoot(tempRoot, root, opts.Overwrite); err != nil {
return Manifest{}, err
}
removeTemp = false
return manifest, nil
}
func cleanBundleRoot(root string) (string, error) {
if root == "" {
return "", fmt.Errorf("root is required")
}
return filepath.Clean(root), nil
}
func copyBundleFiles(root string, files []BundleFile) ([]string, error) {
paths := make([]string, 0, len(files))
seen := make(map[string]struct{}, len(files))
for index, file := range files {
if file.SourcePath == "" {
return nil, fmt.Errorf("files[%d].source_path is required", index)
}
if err := ValidateSourcePath(file.Path); err != nil {
return nil, fmt.Errorf("files[%d].path: %w", index, err)
}
if _, exists := seen[file.Path]; exists {
return nil, fmt.Errorf("files[%d].path duplicates %q", index, file.Path)
}
seen[file.Path] = struct{}{}
if err := copyBundleFile(root, file); err != nil {
return nil, fmt.Errorf("files[%d]: %w", index, err)
}
paths = append(paths, file.Path)
}
return paths, nil
}
func copyBundleFile(root string, file BundleFile) error {
info, err := os.Lstat(file.SourcePath)
if err != nil {
return fmt.Errorf("stat source %q: %w", file.SourcePath, err)
}
if !info.Mode().IsRegular() {
return fmt.Errorf("source %q must be a regular file", file.SourcePath)
}
destination := filepath.Join(root, filepath.FromSlash(file.Path))
if err := os.MkdirAll(filepath.Dir(destination), 0o755); err != nil {
return fmt.Errorf("create destination directory: %w", err)
}
source, err := os.Open(file.SourcePath)
if err != nil {
return fmt.Errorf("open source %q: %w", file.SourcePath, err)
}
defer source.Close()
mode := info.Mode().Perm()
if mode == 0 {
mode = 0o600
}
target, err := os.OpenFile(destination, os.O_WRONLY|os.O_CREATE|os.O_EXCL, mode)
if err != nil {
return fmt.Errorf("create destination %q: %w", file.Path, err)
}
if _, err := io.Copy(target, source); err != nil {
_ = target.Close()
return fmt.Errorf("copy to destination %q: %w", file.Path, err)
}
if err := target.Close(); err != nil {
return fmt.Errorf("close destination %q: %w", file.Path, err)
}
return nil
}
func promoteBundleRoot(tempRoot, root string, overwrite bool) error {
if !overwrite {
if _, err := os.Lstat(root); err == nil {
return fmt.Errorf("bundle root %q already exists", root)
} else if !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("stat bundle root: %w", err)
}
if err := os.Rename(tempRoot, root); err != nil {
return fmt.Errorf("promote bundle root: %w", err)
}
return nil
}
if _, err := os.Lstat(root); errors.Is(err, os.ErrNotExist) {
if err := os.Rename(tempRoot, root); err != nil {
return fmt.Errorf("promote bundle root: %w", err)
}
return nil
} else if err != nil {
return fmt.Errorf("stat bundle root: %w", err)
}
backupRoot, err := reserveSiblingPath(filepath.Dir(root), "."+filepath.Base(root)+"-backup-*.tmp")
if err != nil {
return err
}
if err := os.Rename(root, backupRoot); err != nil {
return fmt.Errorf("move existing bundle root: %w", err)
}
if err := os.Rename(tempRoot, root); err != nil {
restoreErr := os.Rename(backupRoot, root)
if restoreErr != nil {
return fmt.Errorf("promote bundle root: %w; restore existing bundle root: %v", err, restoreErr)
}
return fmt.Errorf("promote bundle root: %w", err)
}
_ = os.RemoveAll(backupRoot)
return nil
}
func reserveSiblingPath(parent, pattern string) (string, error) {
path, err := os.MkdirTemp(parent, pattern)
if err != nil {
return "", fmt.Errorf("reserve backup path: %w", err)
}
if err := os.Remove(path); err != nil {
return "", fmt.Errorf("reserve backup path: %w", err)
}
return path, nil
}

View File

@@ -0,0 +1,38 @@
package bundle_test
import (
"bytes"
"context"
"os"
"path/filepath"
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/app"
"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
)
func TestWriteBundleOutputValidatesThroughDistributor(t *testing.T) {
sourceRoot := t.TempDir()
outputRoot := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(filepath.Join(sourceRoot, "report.md"), []byte("# Report\n"), 0o600); err != nil {
t.Fatalf("write source: %v", err)
}
if _, err := bundle.WriteBundle(bundle.WriteBundleOptions{
Root: outputRoot,
ID: "reports.distributor",
Files: []bundle.BundleFile{
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
},
}); err != nil {
t.Fatalf("WriteBundle() error = %v", err)
}
var stdout bytes.Buffer
if err := app.Validate(context.Background(), app.ValidateOptions{Path: outputRoot, Stdout: &stdout}); err != nil {
t.Fatalf("app.Validate() error = %v", err)
}
if got, want := stdout.String(), "Validated 1 bundle(s)\n"; got != want {
t.Fatalf("stdout = %q, want %q", got, want)
}
}

232
pkg/bundle/writer_test.go Normal file
View File

@@ -0,0 +1,232 @@
package bundle
import (
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
func TestWriteBundleCreatesCompleteBundle(t *testing.T) {
sourceRoot := t.TempDir()
outputRoot := filepath.Join(t.TempDir(), "bundle")
writeFile(t, sourceRoot, "report.md", "# Report\n")
writeFile(t, sourceRoot, "summary.txt", "Summary\n")
created := time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC)
manifest, err := WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.writer",
Created: created,
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "summary.txt"), Path: "nested/summary.txt"},
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
},
})
if err != nil {
t.Fatalf("WriteBundle() error = %v", err)
}
if !manifest.Created.Equal(created) {
t.Fatalf("created = %s, want %s", manifest.Created, created)
}
if paths := manifestPaths(manifest); !reflect.DeepEqual(paths, []string{"nested/summary.txt", "report.md"}) {
t.Fatalf("paths = %v, want caller order", paths)
}
if got := readFile(t, outputRoot, "nested/summary.txt"); got != "Summary\n" {
t.Fatalf("nested summary = %q", got)
}
if got := readFile(t, outputRoot, "report.md"); got != "# Report\n" {
t.Fatalf("report = %q", got)
}
loaded, err := LoadManifest(outputRoot)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if !reflect.DeepEqual(loaded, manifest) {
t.Fatalf("loaded manifest = %#v, want %#v", loaded, manifest)
}
if err := ValidateBundle(outputRoot, loaded); err != nil {
t.Fatalf("ValidateBundle() error = %v", err)
}
}
func TestWriteBundleDefaultsCreated(t *testing.T) {
sourceRoot := t.TempDir()
outputRoot := filepath.Join(t.TempDir(), "bundle")
writeFile(t, sourceRoot, "report.md", "# Report\n")
before := time.Now().UTC()
manifest, err := WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.created",
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
},
})
if err != nil {
t.Fatalf("WriteBundle() error = %v", err)
}
after := time.Now().UTC()
if manifest.Created.Before(before) || manifest.Created.After(after) {
t.Fatalf("created = %s, want between %s and %s", manifest.Created, before, after)
}
}
func TestWriteBundleRejectsDuplicatePaths(t *testing.T) {
sourceRoot := t.TempDir()
writeFile(t, sourceRoot, "report.md", "# Report\n")
_, err := WriteBundle(WriteBundleOptions{
Root: filepath.Join(t.TempDir(), "bundle"),
ID: "reports.duplicate",
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
},
})
if err == nil || !strings.Contains(err.Error(), "duplicates") {
t.Fatalf("WriteBundle() error = %v, want duplicate path error", err)
}
}
func TestWriteBundleRejectsSymlinkSource(t *testing.T) {
sourceRoot := t.TempDir()
writeFile(t, sourceRoot, "target.md", "# Report\n")
linkPath := filepath.Join(sourceRoot, "link.md")
if err := os.Symlink("target.md", linkPath); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
_, err := WriteBundle(WriteBundleOptions{
Root: filepath.Join(t.TempDir(), "bundle"),
ID: "reports.symlink",
Files: []BundleFile{
{SourcePath: linkPath, Path: "report.md"},
},
})
if err == nil || !strings.Contains(err.Error(), "regular file") {
t.Fatalf("WriteBundle() error = %v, want regular file error", err)
}
}
func TestWriteBundleDoesNotOverwriteByDefault(t *testing.T) {
sourceRoot := t.TempDir()
outputRoot := filepath.Join(t.TempDir(), "bundle")
writeFile(t, sourceRoot, "report.md", "old\n")
if _, err := WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.old",
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
},
}); err != nil {
t.Fatalf("initial WriteBundle() error = %v", err)
}
writeFile(t, sourceRoot, "report.md", "new\n")
_, err := WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.new",
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
},
})
if err == nil || !strings.Contains(err.Error(), "already exists") {
t.Fatalf("WriteBundle() error = %v, want exists error", err)
}
if got := readFile(t, outputRoot, "report.md"); got != "old\n" {
t.Fatalf("report = %q, want old content", got)
}
}
func TestWriteBundleOverwritesExistingRoot(t *testing.T) {
sourceRoot := t.TempDir()
outputRoot := filepath.Join(t.TempDir(), "bundle")
writeFile(t, sourceRoot, "old.md", "old\n")
if _, err := WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.old",
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "old.md"), Path: "old.md"},
},
}); err != nil {
t.Fatalf("initial WriteBundle() error = %v", err)
}
writeFile(t, sourceRoot, "new.md", "new\n")
manifest, err := WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.new",
Overwrite: true,
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "new.md"), Path: "new.md"},
},
})
if err != nil {
t.Fatalf("WriteBundle(overwrite) error = %v", err)
}
if manifest.ID != "reports.new" {
t.Fatalf("manifest id = %q, want reports.new", manifest.ID)
}
if _, err := os.Stat(filepath.Join(outputRoot, "old.md")); !os.IsNotExist(err) {
t.Fatalf("old file stat error = %v, want not exist", err)
}
if got := readFile(t, outputRoot, "new.md"); got != "new\n" {
t.Fatalf("new file = %q", got)
}
if err := ValidateBundle(outputRoot, manifest); err != nil {
t.Fatalf("ValidateBundle() error = %v", err)
}
}
func TestWriteBundleKeepsExistingRootWhenReplacementBuildFails(t *testing.T) {
sourceRoot := t.TempDir()
outputRoot := filepath.Join(t.TempDir(), "bundle")
writeFile(t, sourceRoot, "report.md", "old\n")
oldManifest, err := WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.old",
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "report.md"), Path: "report.md"},
},
})
if err != nil {
t.Fatalf("initial WriteBundle() error = %v", err)
}
_, err = WriteBundle(WriteBundleOptions{
Root: outputRoot,
ID: "reports.new",
Overwrite: true,
Files: []BundleFile{
{SourcePath: filepath.Join(sourceRoot, "missing.md"), Path: "report.md"},
},
})
if err == nil {
t.Fatal("WriteBundle(overwrite) error = nil, want missing source error")
}
loaded, err := LoadManifest(outputRoot)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if !reflect.DeepEqual(loaded, oldManifest) {
t.Fatalf("loaded manifest = %#v, want old manifest %#v", loaded, oldManifest)
}
if got := readFile(t, outputRoot, "report.md"); got != "old\n" {
t.Fatalf("report = %q, want old content", got)
}
}
func readFile(t *testing.T, root, relative string) string {
t.Helper()
data, err := os.ReadFile(filepath.Join(root, filepath.FromSlash(relative)))
if err != nil {
t.Fatalf("read %s: %v", relative, err)
}
return string(data)
}