Add local source publication

This commit is contained in:
2026-05-31 02:21:36 +00:00
parent aeee91940d
commit e296361042
17 changed files with 1019 additions and 84 deletions

View File

@@ -3,10 +3,10 @@
## Shortest useful command
```sh
go run ./cmd/distributor validate examples/source-bundle
go run ./cmd/distributor run --config examples/local-publish.yml
```
This validates a local source bundle fixture.
This validates and publishes the example source bundle to `workspace/published/source-bundle`.
## Command overview
@@ -20,13 +20,15 @@ distributor inspect
`version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time.
`run --config <path> --dry-run` loads and validates configuration, then prints a concise summary of configured pipelines and destinations. It does not discover bundles or publish files yet.
`run --config <path>` executes configured local-to-local pipelines that publish source files only.
`run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions without writing files.
`validate <path>` validates a local source bundle directory or a local tree containing source bundles.
`inspect <path>` validates discovered local source bundles and prints a concise normalized summary.
`run` without `--dry-run` intentionally fails with a clear `not implemented` error until execution behavior exists.
Remote backends and HTML publication are not implemented yet.
## Flag reference
@@ -57,8 +59,14 @@ Inspect a source bundle:
go run ./cmd/distributor inspect examples/source-bundle
```
Validate a config file without publishing:
Preview local publication without writing:
```sh
go run ./cmd/distributor run --config examples/local-to-local.yml --dry-run
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
```
Publish the local example:
```sh
go run ./cmd/distributor run --config examples/local-publish.yml
```

View File

@@ -2,15 +2,15 @@
## Config file location
`distributor run --config <path> --dry-run` loads the YAML config at the path provided by `--config`.
`distributor run --config <path>` loads the YAML config at the path provided by `--config`.
If `--config` is omitted during dry-run, the built-in default path is:
If `--config` is omitted during run, the built-in default path is:
```text
/usr/local/etc/distributor/config.yml
```
The current implementation loads and validates configuration only. Bundle discovery and publication are not implemented yet.
The current implementation supports local-to-local source-file publication. Remote backends and HTML publication are not implemented yet.
## Minimal config
@@ -40,18 +40,16 @@ pipelines:
on_digest_mismatch: fail
destinations:
- id: archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
backend: local
path: /srv/reports/archive
publish:
source: true
html: false
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
## Reference

29
docs/internal/publish.md Normal file
View File

@@ -0,0 +1,29 @@
# Publish
## Purpose
`internal/publish` plans and executes publication for one validated source bundle and one destination.
## Inputs and outputs
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transfer policy, destination bundle path, and existing destination state.
Output is a plan with an action, reason, and selected source outputs. Execution writes selected source files and `.distributor.json` for publish or replacement actions.
## Actions
Supported actions are publish new, replace older destination, skip same source, skip newer destination, fail conflict, and fail unmanaged destination.
## Boundaries
The current implementation publishes source files only. HTML generation and remote backend execution are not implemented.
The package uses `internal/state` for destination comparison and `internal/storage` for IO. It does not parse CLI flags or load config files.
## Safety
Replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Failed local writes trigger cleanup of outputs written during the failed attempt.
## Tests
Before changing publish behavior, inspect tests under `internal/publish` and local run tests under `internal/app`.

39
docs/operations.md Normal file
View File

@@ -0,0 +1,39 @@
# Distributor Operations
## Normal workflow
Preview a local publication:
```sh
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
```
Run the local publication:
```sh
go run ./cmd/distributor run --config examples/local-publish.yml
```
## Filesystem layout
Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root.
The maintained example writes under `workspace/`, which is ignored by Git.
## Destination state
Each published destination bundle contains `.distributor.json`. This state file records the source manifest and copied source outputs. It is the authoritative marker that a destination path is managed by `distributor`.
`manifest.json` from the source bundle is not copied as destination state.
## Retry behavior
If a destination already has matching `.distributor.json`, publication skips it as already published.
If destination state is older than the source manifest, publication replaces only managed outputs recorded in `.distributor.json` plus the state file.
If a write fails during local publication, `distributor` removes outputs written during that failed attempt where possible so a retry does not see an unmanaged destination.
## Caveats
Only local-to-local source-file publication is implemented. SSH, S3, HTML generation, notification, and force overwrite behavior are not implemented.