Add local source publication
This commit is contained in:
20
docs/cli.md
20
docs/cli.md
@@ -3,10 +3,10 @@
|
||||
## Shortest useful command
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate examples/source-bundle
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||
```
|
||||
|
||||
This validates a local source bundle fixture.
|
||||
This validates and publishes the example source bundle to `workspace/published/source-bundle`.
|
||||
|
||||
## Command overview
|
||||
|
||||
@@ -20,13 +20,15 @@ distributor inspect
|
||||
|
||||
`version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time.
|
||||
|
||||
`run --config <path> --dry-run` loads and validates configuration, then prints a concise summary of configured pipelines and destinations. It does not discover bundles or publish files yet.
|
||||
`run --config <path>` executes configured local-to-local pipelines that publish source files only.
|
||||
|
||||
`run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions without writing files.
|
||||
|
||||
`validate <path>` validates a local source bundle directory or a local tree containing source bundles.
|
||||
|
||||
`inspect <path>` validates discovered local source bundles and prints a concise normalized summary.
|
||||
|
||||
`run` without `--dry-run` intentionally fails with a clear `not implemented` error until execution behavior exists.
|
||||
Remote backends and HTML publication are not implemented yet.
|
||||
|
||||
## Flag reference
|
||||
|
||||
@@ -57,8 +59,14 @@ Inspect a source bundle:
|
||||
go run ./cmd/distributor inspect examples/source-bundle
|
||||
```
|
||||
|
||||
Validate a config file without publishing:
|
||||
Preview local publication without writing:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/local-to-local.yml --dry-run
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
|
||||
```
|
||||
|
||||
Publish the local example:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||
```
|
||||
|
||||
@@ -2,15 +2,15 @@
|
||||
|
||||
## Config file location
|
||||
|
||||
`distributor run --config <path> --dry-run` loads the YAML config at the path provided by `--config`.
|
||||
`distributor run --config <path>` loads the YAML config at the path provided by `--config`.
|
||||
|
||||
If `--config` is omitted during dry-run, the built-in default path is:
|
||||
If `--config` is omitted during run, the built-in default path is:
|
||||
|
||||
```text
|
||||
/usr/local/etc/distributor/config.yml
|
||||
```
|
||||
|
||||
The current implementation loads and validates configuration only. Bundle discovery and publication are not implemented yet.
|
||||
The current implementation supports local-to-local source-file publication. Remote backends and HTML publication are not implemented yet.
|
||||
|
||||
## Minimal config
|
||||
|
||||
@@ -40,18 +40,16 @@ pipelines:
|
||||
on_digest_mismatch: fail
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
backend: local
|
||||
path: /srv/reports/archive
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
## Reference
|
||||
|
||||
29
docs/internal/publish.md
Normal file
29
docs/internal/publish.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Publish
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/publish` plans and executes publication for one validated source bundle and one destination.
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transfer policy, destination bundle path, and existing destination state.
|
||||
|
||||
Output is a plan with an action, reason, and selected source outputs. Execution writes selected source files and `.distributor.json` for publish or replacement actions.
|
||||
|
||||
## Actions
|
||||
|
||||
Supported actions are publish new, replace older destination, skip same source, skip newer destination, fail conflict, and fail unmanaged destination.
|
||||
|
||||
## Boundaries
|
||||
|
||||
The current implementation publishes source files only. HTML generation and remote backend execution are not implemented.
|
||||
|
||||
The package uses `internal/state` for destination comparison and `internal/storage` for IO. It does not parse CLI flags or load config files.
|
||||
|
||||
## Safety
|
||||
|
||||
Replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Failed local writes trigger cleanup of outputs written during the failed attempt.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing publish behavior, inspect tests under `internal/publish` and local run tests under `internal/app`.
|
||||
39
docs/operations.md
Normal file
39
docs/operations.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# Distributor Operations
|
||||
|
||||
## Normal workflow
|
||||
|
||||
Preview a local publication:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
|
||||
```
|
||||
|
||||
Run the local publication:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||
```
|
||||
|
||||
## Filesystem layout
|
||||
|
||||
Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root.
|
||||
|
||||
The maintained example writes under `workspace/`, which is ignored by Git.
|
||||
|
||||
## Destination state
|
||||
|
||||
Each published destination bundle contains `.distributor.json`. This state file records the source manifest and copied source outputs. It is the authoritative marker that a destination path is managed by `distributor`.
|
||||
|
||||
`manifest.json` from the source bundle is not copied as destination state.
|
||||
|
||||
## Retry behavior
|
||||
|
||||
If a destination already has matching `.distributor.json`, publication skips it as already published.
|
||||
|
||||
If destination state is older than the source manifest, publication replaces only managed outputs recorded in `.distributor.json` plus the state file.
|
||||
|
||||
If a write fails during local publication, `distributor` removes outputs written during that failed attempt where possible so a retry does not see an unmanaged destination.
|
||||
|
||||
## Caveats
|
||||
|
||||
Only local-to-local source-file publication is implemented. SSH, S3, HTML generation, notification, and force overwrite behavior are not implemented.
|
||||
Reference in New Issue
Block a user