Update roadmap documentation to remove completed work
This commit is contained in:
@@ -1,580 +0,0 @@
|
||||
# Distributor Configuration Roadmap
|
||||
|
||||
This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.
|
||||
|
||||
## Configuration Goals
|
||||
|
||||
The MVP configuration should be:
|
||||
|
||||
- explicit enough to avoid hidden publication behavior;
|
||||
- compact enough for routine self-hosted use;
|
||||
- backend-agnostic at the pipeline layer;
|
||||
- capable of local, SSH/SFTP, and S3-compatible source and destination backends;
|
||||
- ready for future notification adapters without exposing a fake notification feature in the MVP.
|
||||
|
||||
## Top-Level Shape
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
## Pipeline Fields
|
||||
|
||||
Each pipeline must include:
|
||||
|
||||
- `id`: Required stable pipeline identifier.
|
||||
- `source`: Required source backend configuration.
|
||||
- `destinations`: Required non-empty list of destination configurations.
|
||||
|
||||
Optional pipeline-level fields:
|
||||
|
||||
- `validation`: Source validation behavior.
|
||||
- Future: `notifications` or `notify`, when notification adapters are implemented.
|
||||
|
||||
A pipeline has exactly one source and one or more destinations.
|
||||
|
||||
## Pipeline ID Rules
|
||||
|
||||
`pipelines[].id` should:
|
||||
|
||||
- be required;
|
||||
- be unique across the config file;
|
||||
- be stable over time;
|
||||
- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`.
|
||||
|
||||
Recommended validation:
|
||||
|
||||
```text
|
||||
^[a-zA-Z0-9][a-zA-Z0-9._-]*$
|
||||
```
|
||||
|
||||
## Source Configuration
|
||||
|
||||
`source` defines the source root where bundles are discovered.
|
||||
|
||||
The source backend may be:
|
||||
|
||||
- `local`;
|
||||
- `ssh`;
|
||||
- `s3`.
|
||||
|
||||
The source is scanned for `manifest.json` files beneath the configured root.
|
||||
|
||||
### Local source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: local`
|
||||
- `path`
|
||||
|
||||
### SSH source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: ssh
|
||||
uri: ssh://reports@example.com:22
|
||||
path: /var/spool/distributor/weather
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: ssh`
|
||||
- `uri`
|
||||
- `path`
|
||||
|
||||
Recommended authentication behavior:
|
||||
|
||||
- use SSH agent by default;
|
||||
- use local known_hosts validation by default;
|
||||
- support optional key file configuration later if needed;
|
||||
- do not require passwords in YAML.
|
||||
|
||||
Optional future fields:
|
||||
|
||||
```yaml
|
||||
known_hosts: /home/user/.ssh/known_hosts
|
||||
key_file: /home/user/.ssh/id_ed25519
|
||||
```
|
||||
|
||||
### S3 source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: incoming/weather
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: s3`
|
||||
- `endpoint`
|
||||
- `bucket`
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `prefix`
|
||||
- `region`
|
||||
- `force_path_style`
|
||||
- `credentials`
|
||||
|
||||
Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.
|
||||
|
||||
## Destination Configuration
|
||||
|
||||
Each destination defines one publication target for a pipeline.
|
||||
|
||||
Required destination fields:
|
||||
|
||||
- `id`
|
||||
- `backend`
|
||||
- backend-specific location fields;
|
||||
- `publish`
|
||||
|
||||
Optional destination fields:
|
||||
|
||||
- `transform`
|
||||
- `transfer`
|
||||
|
||||
Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.
|
||||
|
||||
## Destination ID Rules
|
||||
|
||||
`destinations[].id` should:
|
||||
|
||||
- be required;
|
||||
- be unique within the containing pipeline;
|
||||
- be stable over time;
|
||||
- use a slug-like format.
|
||||
|
||||
Recommended examples:
|
||||
|
||||
- `markdown-archive`
|
||||
- `static-site`
|
||||
- `full-mirror`
|
||||
|
||||
## Local Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: local-static
|
||||
backend: local
|
||||
path: /srv/www/reports
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: local`
|
||||
- `path`
|
||||
- `publish`
|
||||
|
||||
## SSH Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: ssh`
|
||||
- `uri`
|
||||
- `path`
|
||||
- `publish`
|
||||
|
||||
The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`.
|
||||
|
||||
## S3 Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: s3`
|
||||
- `endpoint`
|
||||
- `bucket`
|
||||
- `publish`
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `prefix`
|
||||
- `region`
|
||||
- `force_path_style`
|
||||
- `credentials`
|
||||
|
||||
## Backend Configuration Normalization
|
||||
|
||||
The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.
|
||||
|
||||
Validation should catch:
|
||||
|
||||
- missing backend names;
|
||||
- unsupported backend names;
|
||||
- missing backend-specific required fields;
|
||||
- duplicate pipeline IDs;
|
||||
- duplicate destination IDs within a pipeline;
|
||||
- empty destination lists;
|
||||
- invalid policy values.
|
||||
|
||||
## Publication Policy
|
||||
|
||||
`publish` controls which categories of files are written to a destination.
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `source`: Publish source artifacts listed in `manifest.json`.
|
||||
- `html`: Publish HTML files generated from Markdown source artifacts.
|
||||
|
||||
At least one of `source` or `html` must be true.
|
||||
|
||||
Recommended defaults:
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP.
|
||||
|
||||
## Transform Configuration
|
||||
|
||||
For MVP, the only supported transform is Markdown to HTML.
|
||||
|
||||
```yaml
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `enabled`: Whether Markdown-to-HTML transform is enabled.
|
||||
- `mode`: Output mode. MVP value: `sidecar`.
|
||||
|
||||
MVP `sidecar` behavior:
|
||||
|
||||
- each listed Markdown source file generates an HTML file with the same base path and `.html` extension;
|
||||
- `report.md` generates `report.html`;
|
||||
- generated files are destination publication artifacts;
|
||||
- source bundles are not mutated.
|
||||
|
||||
MVP defaulting:
|
||||
|
||||
- If `publish.html` is false, transform may be omitted.
|
||||
- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail.
|
||||
|
||||
## Validation Policy
|
||||
|
||||
Pipeline-level validation is intentionally narrow in the MVP.
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
```
|
||||
|
||||
Supported value:
|
||||
|
||||
- `fail`
|
||||
|
||||
Default:
|
||||
|
||||
```yaml
|
||||
on_digest_mismatch: fail
|
||||
```
|
||||
|
||||
Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.
|
||||
|
||||
## Transfer Policy
|
||||
|
||||
Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path.
|
||||
|
||||
```yaml
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
Supported fields:
|
||||
|
||||
- `on_destination_same`
|
||||
- `on_destination_older`
|
||||
- `on_destination_newer`
|
||||
- `on_conflict`
|
||||
|
||||
MVP supported values are intentionally limited by field:
|
||||
|
||||
- `on_destination_same`: `skip` or `fail`
|
||||
- `on_destination_older`: `replace` or `fail`
|
||||
- `on_destination_newer`: `skip` or `fail`
|
||||
- `on_conflict`: `fail`
|
||||
|
||||
Recommended MVP defaults:
|
||||
|
||||
```yaml
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
Safety rule:
|
||||
|
||||
- `replace` must never perform broad deletion against a destination root.
|
||||
- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
|
||||
- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
|
||||
- Broader replacement values, including replacing newer destinations or conflicts, are deferred to a later explicit force-overwrite stage.
|
||||
|
||||
## Path Mapping
|
||||
|
||||
MVP path mapping is fixed:
|
||||
|
||||
```text
|
||||
destination bundle path = destination root + source relative bundle path
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
source root: /var/spool/reports
|
||||
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
|
||||
relative bundle path: weather/daily/brentwood/2026-05-30
|
||||
|
||||
destination root: /srv/www/reports
|
||||
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30
|
||||
```
|
||||
|
||||
Future config may support explicit path mapping, but MVP should not.
|
||||
|
||||
## Dry Run Configuration and CLI Behavior
|
||||
|
||||
Dry-run should be a CLI flag rather than a persistent config setting.
|
||||
|
||||
```bash
|
||||
distributor run --config config.yml --dry-run
|
||||
```
|
||||
|
||||
Dry-run should report:
|
||||
|
||||
- pipeline ID;
|
||||
- source backend;
|
||||
- destination ID;
|
||||
- destination backend;
|
||||
- discovered bundle ID;
|
||||
- relative bundle path;
|
||||
- planned action;
|
||||
- reason;
|
||||
- transform outputs that would be generated;
|
||||
- files that would be written or deleted.
|
||||
|
||||
## Example: Weather Pipeline
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@web.example.com:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
## Example: D&D Recap Pipeline
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: dnd-session-recaps
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/dnd/session-recaps
|
||||
|
||||
destinations:
|
||||
- id: private-markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: dnd/session-recaps
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
- id: private-html-site
|
||||
backend: local
|
||||
path: /srv/www/private/dnd/session-recaps
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
## Future Notification Configuration
|
||||
|
||||
Notification should not be exposed as a functional MVP feature unless an adapter exists.
|
||||
|
||||
The internal pipeline may include a no-op notification stage. Future config may look like:
|
||||
|
||||
```yaml
|
||||
notifications:
|
||||
- id: weather-email
|
||||
backend: email
|
||||
after_destinations:
|
||||
- static-site
|
||||
subject: "Weather report published"
|
||||
```
|
||||
|
||||
Future notification policy should require:
|
||||
|
||||
- notification after successful relevant publication;
|
||||
- idempotency by source manifest id and digest;
|
||||
- no duplicate notification unless explicitly forced.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
|
||||
2. Implement config loading and strict validation.
|
||||
3. Implement backend-specific config validation for local, SSH, and S3.
|
||||
4. Implement defaulting for validation and transfer policies.
|
||||
5. Require explicit transform configuration when `publish.html` is true.
|
||||
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
|
||||
7. Connect config to backend registry and publish planner.
|
||||
8. Add `--pipeline` filtering for targeted runs.
|
||||
9. Add `--dry-run` output that reflects the resolved config and planned actions.
|
||||
|
||||
## Deferred Configuration
|
||||
|
||||
The following configuration ideas are intentionally outside the MVP:
|
||||
|
||||
- warning-only digest mismatch handling;
|
||||
- unmanaged destination overwrite flags until the explicit force-overwrite roadmap stage;
|
||||
- force replacement of destinations with different source ids until the explicit force-overwrite roadmap stage.
|
||||
@@ -1,398 +0,0 @@
|
||||
# Distributor Contracts Roadmap
|
||||
|
||||
This roadmap defines the contracts that `distributor` should implement before or alongside the MVP. The goal is to make bundle validation, destination state, digest verification, and safe replacement deterministic and testable before backend-specific publication behavior is layered on top.
|
||||
|
||||
## Purpose
|
||||
|
||||
`distributor` publishes manifested report bundles produced by other applications. Producer applications own domain-specific report generation. `distributor` owns validation, optional transformation, destination publication, and destination state.
|
||||
|
||||
The MVP contract has two durable files:
|
||||
|
||||
- `manifest.json`: source-owned bundle manifest produced by the upstream application.
|
||||
- `.distributor.json`: destination-owned publication state written by `distributor`.
|
||||
|
||||
`manifest.json` is not copied to the destination as destination state. Instead, `.distributor.json` records the normalized source manifest, generated output metadata, and distributor-owned publication metadata.
|
||||
|
||||
All manifest and state timestamps should be serialized as RFC3339. Internal comparison should use parsed timestamp values, and distributor-written timestamps should be normalized to RFC3339 UTC.
|
||||
|
||||
## Terminology
|
||||
|
||||
- **Source root**: Configured root path for a pipeline source.
|
||||
- **Bundle root**: Directory beneath the source root that contains `manifest.json`.
|
||||
- **Relative bundle path**: Bundle root path relative to the source root.
|
||||
- **Destination root**: Configured root path or prefix for a destination.
|
||||
- **Destination bundle path**: Destination root plus the relative bundle path, unless a future mapping option overrides that behavior.
|
||||
- **Source artifact**: File listed in the source `manifest.json`.
|
||||
- **Generated artifact**: File created by `distributor`, such as an HTML file derived from Markdown.
|
||||
- **Destination state**: `.distributor.json` at the destination bundle path.
|
||||
|
||||
## Source Bundle Contract
|
||||
|
||||
A source bundle is a directory containing a `manifest.json` file. For MVP, the manifest schema is intentionally minimal.
|
||||
|
||||
### Required `manifest.json` fields
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"id": "weather.daily.brentwood.2026-05-30",
|
||||
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"created": "2026-05-30T11:10:00Z",
|
||||
"files": [
|
||||
{
|
||||
"path": "report.md",
|
||||
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"size": 12345
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required top-level fields:
|
||||
|
||||
- `schema_version`: Source manifest schema version. MVP value: `1`.
|
||||
- `id`: Stable bundle identifier. Required, non-empty string.
|
||||
- `digest`: Bundle digest. Required, lowercase `sha256:<64 hex>` string.
|
||||
- `created`: Bundle creation timestamp. Required RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
|
||||
- `files`: Non-empty array of file objects.
|
||||
|
||||
Required file fields:
|
||||
|
||||
- `path`: Relative path from bundle root to source artifact.
|
||||
- `sha256`: Per-file digest as lowercase `sha256:<64 hex>`.
|
||||
- `size`: File size in bytes.
|
||||
|
||||
No other source manifest fields are required for the MVP. Additional fields may be ignored unless later documented. Destination state records the normalized source manifest model, not raw unknown manifest fields.
|
||||
|
||||
## Source Path Safety Rules
|
||||
|
||||
For every `files[].path`:
|
||||
|
||||
- Path must be relative.
|
||||
- Path must not be empty.
|
||||
- Path must not contain `..` segments.
|
||||
- Path must not resolve outside the bundle root.
|
||||
- Path must use slash-separated logical paths in the manifest.
|
||||
- Absolute paths are invalid.
|
||||
- Symlinks should be rejected for MVP unless a later policy deliberately supports them.
|
||||
- `manifest.json` itself should not be listed as a source artifact.
|
||||
- `.distributor.json` should not be listed as a source artifact.
|
||||
- Duplicate logical file paths are invalid after path normalization.
|
||||
|
||||
The implementation should validate paths before reading file contents.
|
||||
|
||||
## Digest Contract
|
||||
|
||||
The MVP validates both per-file digests and the bundle digest.
|
||||
|
||||
### Per-file digest
|
||||
|
||||
For each file listed in `files`, compute:
|
||||
|
||||
```text
|
||||
sha256(file bytes)
|
||||
```
|
||||
|
||||
The computed digest must match `files[].sha256`.
|
||||
|
||||
The actual file size must match `files[].size`.
|
||||
|
||||
### Bundle digest
|
||||
|
||||
The bundle digest is computed from the listed file records in the listed order. The canonical algorithm is:
|
||||
|
||||
1. For each file listed in `files`, in order:
|
||||
- validate the file path;
|
||||
- compute the file SHA256;
|
||||
- determine the file size.
|
||||
2. Construct a canonical JSON array containing only:
|
||||
- `path`;
|
||||
- `sha256`;
|
||||
- `size`.
|
||||
3. Preserve the source manifest's file order.
|
||||
4. Encode the array deterministically with:
|
||||
- object fields in exactly this order: `path`, `sha256`, `size`;
|
||||
- no extra spaces;
|
||||
- no trailing newline;
|
||||
- lowercase `sha256:<64 hex>` digest strings.
|
||||
5. Compute `sha256(canonical JSON bytes)`.
|
||||
6. Compare the result to top-level `digest`.
|
||||
|
||||
Conceptual canonical payload:
|
||||
|
||||
```json
|
||||
[
|
||||
{"path":"report.md","sha256":"sha256:...","size":12345},
|
||||
{"path":"summary.txt","sha256":"sha256:...","size":234}
|
||||
]
|
||||
```
|
||||
|
||||
This avoids ambiguous concatenation of file bytes while keeping the manifest small.
|
||||
|
||||
Implementation fixtures should include at least one reference manifest and canonical payload with known per-file and bundle digests.
|
||||
|
||||
## Digest Mismatch Behavior
|
||||
|
||||
Digest validation is always fatal in the MVP. A digest mismatch must fail the affected bundle, pipeline, and run before any destination writes occur.
|
||||
|
||||
Warning-only digest behavior is intentionally deferred until a later roadmap accepts transitional ingestion semantics.
|
||||
|
||||
## Bundle Discovery Contract
|
||||
|
||||
A source path may contain one bundle or a tree of bundles. Discovery should scan beneath the configured source root for `manifest.json` files.
|
||||
|
||||
For each discovered manifest:
|
||||
|
||||
- Bundle root is the directory containing `manifest.json`.
|
||||
- Relative bundle path is computed relative to source root.
|
||||
- Destination bundle path is destination root plus relative bundle path, unless a future mapping option overrides it.
|
||||
|
||||
If nested manifests are found, the MVP should fail with a clear error unless a later policy defines nested-bundle semantics.
|
||||
|
||||
## Destination State Contract
|
||||
|
||||
Destinations are managed by `.distributor.json`, not by copying `manifest.json`.
|
||||
|
||||
A destination bundle path is considered distributor-managed only when it contains a valid `.distributor.json` written by `distributor`. Force or unmanaged-overwrite behavior is not part of the MVP.
|
||||
|
||||
### Required `.distributor.json` shape
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"distributor_version": "0.1.0",
|
||||
"pipeline_id": "weather-daily",
|
||||
"destination_id": "static-site",
|
||||
"published_at": "2026-05-30T11:12:00Z",
|
||||
"source": {
|
||||
"manifest": {
|
||||
"schema_version": 1,
|
||||
"id": "weather.daily.brentwood.2026-05-30",
|
||||
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"created": "2026-05-30T11:10:00Z",
|
||||
"files": [
|
||||
{
|
||||
"path": "report.md",
|
||||
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"size": 12345
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"path": "report.html",
|
||||
"kind": "generated",
|
||||
"source_path": "report.md",
|
||||
"transform": "markdown_to_html",
|
||||
"sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333",
|
||||
"size": 23456
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `schema_version`: Destination state schema version. MVP value: `1`.
|
||||
- `distributor_version`: Optional diagnostic distributor version. It must not affect source comparison.
|
||||
- `pipeline_id`: Pipeline that produced the destination publication.
|
||||
- `destination_id`: Destination within the pipeline.
|
||||
- `published_at`: RFC3339 timestamp.
|
||||
- `source.manifest`: Normalized source manifest model used for this publication.
|
||||
- `outputs`: Array of files written by `distributor` for this destination.
|
||||
|
||||
Required output fields:
|
||||
|
||||
- `path`: Destination-relative output path within the destination bundle path.
|
||||
- `kind`: `source` or `generated`.
|
||||
- `source_path`: Source artifact path that produced this output. For copied source files, this should equal `path` unless renamed by a future feature.
|
||||
- `transform`: Transform identifier for generated files. Empty or omitted may be allowed for copied source files.
|
||||
- `sha256`: Output file digest as lowercase `sha256:<64 hex>`.
|
||||
- `size`: Output file size in bytes.
|
||||
|
||||
## Destination Comparison Rules
|
||||
|
||||
Destination comparison uses `.distributor.json`, not destination `manifest.json`.
|
||||
|
||||
For a source bundle and destination bundle path:
|
||||
|
||||
### No `.distributor.json`
|
||||
|
||||
If no `.distributor.json` exists and the destination path is empty:
|
||||
|
||||
- Publish normally.
|
||||
|
||||
If no `.distributor.json` exists and the destination path is non-empty:
|
||||
|
||||
- Fail as unmanaged content.
|
||||
|
||||
Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix.
|
||||
|
||||
### Pipeline or destination mismatch
|
||||
|
||||
If `.distributor.json` exists but its `pipeline_id` or `destination_id` differs from the current pipeline or destination config:
|
||||
|
||||
- Fail as a conflict.
|
||||
|
||||
### Same source manifest
|
||||
|
||||
If `.distributor.json` exists and `source.manifest` exactly matches the current normalized source manifest:
|
||||
|
||||
- Skip as already published.
|
||||
|
||||
### Same source id, destination older
|
||||
|
||||
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is older than the source `created`:
|
||||
|
||||
- Replace destination contents, subject to replacement safety rules.
|
||||
|
||||
### Same source id, destination newer
|
||||
|
||||
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is newer than the source `created`:
|
||||
|
||||
- Skip and log that destination is newer than source.
|
||||
|
||||
### Same source id and created, different digest
|
||||
|
||||
If `.distributor.json` exists, `source.manifest.id` and `created` match but `digest` differs:
|
||||
|
||||
- Fail as a conflict.
|
||||
|
||||
### Different source id
|
||||
|
||||
If `.distributor.json` exists and `source.manifest.id` differs from the source manifest `id`:
|
||||
|
||||
- Fail as a conflict.
|
||||
|
||||
## Replacement Safety Rules
|
||||
|
||||
Replacement is destructive and must be narrow.
|
||||
|
||||
`distributor` must never perform broad deletion against a configured destination root.
|
||||
|
||||
Replacement may occur only at a resolved destination bundle path when:
|
||||
|
||||
- a valid `.distributor.json` exists at that destination bundle path; and
|
||||
- the state identifies the path as distributor-managed; and
|
||||
- the replacement decision follows the destination comparison rules.
|
||||
|
||||
For MVP, replacement should delete only known managed outputs where practical:
|
||||
|
||||
- files listed in existing `.distributor.json.outputs`;
|
||||
- existing `.distributor.json`;
|
||||
- empty directories created by those files, where applicable for filesystem-like backends.
|
||||
|
||||
For S3, replacement should delete only objects under the destination bundle prefix that are listed in `.distributor.json.outputs` plus `.distributor.json`, unless a later managed-prefix deletion policy is explicitly implemented.
|
||||
|
||||
Before any write, planned destination output paths must be checked for collisions. For example, if `report.md` is copied as a source artifact and Markdown transformation would also generate `report.html`, but `report.html` is already a source artifact or another generated output, planning must fail before writing.
|
||||
|
||||
## Transform Output Contract
|
||||
|
||||
Source files are canonical. Generated files are derived publication artifacts.
|
||||
|
||||
For MVP, the only supported transform is Markdown to HTML.
|
||||
|
||||
Recommended MVP behavior:
|
||||
|
||||
- Transform is configured per destination.
|
||||
- Markdown source files are files listed in `manifest.json` with `.md` extension.
|
||||
- Generated HTML files are sidecars by default.
|
||||
- `report.md` generates `report.html`.
|
||||
- Source bundles are never mutated.
|
||||
- Generated outputs are recorded in `.distributor.json.outputs`.
|
||||
- Raw HTML embedded in Markdown is escaped or disabled by default for the MVP to keep generated output deterministic and conservative.
|
||||
|
||||
Future versions may add templates, `index.html`, CSS assets, email-safe HTML, and per-file transform selection.
|
||||
|
||||
## Destination Output Contract
|
||||
|
||||
Each destination chooses which categories of files it receives.
|
||||
|
||||
MVP categories:
|
||||
|
||||
- `source`: copied source artifacts listed in `manifest.json`.
|
||||
- `html`: generated HTML derived from Markdown source artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- Markdown archive: `source: true`, `html: false`.
|
||||
- Static HTML site: `source: false`, `html: true`.
|
||||
- Full mirror: `source: true`, `html: true`.
|
||||
|
||||
Every file written to the destination must be represented in `.distributor.json.outputs`.
|
||||
|
||||
## Atomicity and Partial Failure
|
||||
|
||||
The MVP should prefer staging and promotion where backend semantics permit it.
|
||||
|
||||
Minimum behavior:
|
||||
|
||||
- Validate source before writing destination files.
|
||||
- Do not write `.distributor.json` until all configured destination outputs are successfully written.
|
||||
- If publication fails before `.distributor.json` is written, the destination must not be treated as successfully published on a later run.
|
||||
- Local publication must use staging or equivalent cleanup behavior so a failed write does not leave a confusing unmanaged destination bundle path.
|
||||
- Later cleanup may remove orphaned files, but MVP correctness should rely on `.distributor.json` as the success marker.
|
||||
|
||||
## Example Source Bundle
|
||||
|
||||
```text
|
||||
weather/daily/brentwood/2026-05-30/
|
||||
manifest.json
|
||||
report.md
|
||||
summary.txt
|
||||
```
|
||||
|
||||
Example manifest:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"id": "weather.daily.brentwood.2026-05-30",
|
||||
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"created": "2026-05-30T11:10:00Z",
|
||||
"files": [
|
||||
{
|
||||
"path": "report.md",
|
||||
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"size": 12345
|
||||
},
|
||||
{
|
||||
"path": "summary.txt",
|
||||
"sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
|
||||
"size": 234
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Example Destination Bundle: HTML Only
|
||||
|
||||
```text
|
||||
weather/daily/brentwood/2026-05-30/
|
||||
report.html
|
||||
.distributor.json
|
||||
```
|
||||
|
||||
## Example Destination Bundle: Source Archive
|
||||
|
||||
```text
|
||||
weather/daily/brentwood/2026-05-30/
|
||||
report.md
|
||||
summary.txt
|
||||
.distributor.json
|
||||
```
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
1. Define Go structs for source manifest and destination state.
|
||||
2. Implement source path validation.
|
||||
3. Implement per-file SHA256 and size validation.
|
||||
4. Implement canonical bundle digest validation.
|
||||
5. Implement source bundle discovery beneath a source root.
|
||||
6. Implement `.distributor.json` parsing and validation.
|
||||
7. Implement destination comparison rules.
|
||||
8. Implement replacement safety checks.
|
||||
9. Add fixture bundles for valid, invalid, duplicate path, older, newer, same, and conflict scenarios.
|
||||
10. Add reference canonical digest fixtures.
|
||||
11. Use the contract layer from the publish pipeline and backend adapters.
|
||||
@@ -1,468 +0,0 @@
|
||||
# Documentation Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines a documentation-only refresh plan for `distributor` after the local MVP and cleanup roadmap implementation.
|
||||
|
||||
The goal is to make current-behavior documentation concise, accurate, and compliant with `docs/policy/documentation.md` while keeping planned, aspirational, or unimplemented behavior under `docs/roadmap/`.
|
||||
|
||||
This file is written for an LLM coding agent that will implement the documentation refresh in stages. It does not itself rewrite current user, policy, internal, or example documentation.
|
||||
|
||||
## Repository Documentation Inventory
|
||||
|
||||
The current documentation and examples reviewed are:
|
||||
|
||||
| Path | Current status | Notes |
|
||||
| --- | --- | --- |
|
||||
| `README.md` | Keep and lightly update | Short, accurate orientation page with a runnable local example command. |
|
||||
| `docs/cli.md` | Keep and update against CLI tests | Covers `version`, `run`, `validate`, and `inspect`; should be checked against command help and parser tests. |
|
||||
| `docs/config.md` | Keep and tighten | Current config reference documents local execution and also accepted SSH/S3 config fields. It must clearly distinguish config validation from executable backend support. |
|
||||
| `docs/operations.md` | Keep and expand slightly | Covers local workflow, destination state, retry behavior, cleanup, fan-out failure handling, and caveats. |
|
||||
| `docs/troubleshooting.md` | Added | Covers recurring local MVP failure modes. |
|
||||
| `docs/policy/architecture.md` | Keep and clarify where needed | Development policy is broad and includes future adapter direction. Wording should not imply SSH/S3 adapters currently exist. |
|
||||
| `docs/policy/development.md` | Updated | Contains contributor and agent workflow guidance. |
|
||||
| `docs/policy/documentation.md` | Keep | Canonical documentation policy. No change required unless the policy itself changes. |
|
||||
| `docs/internal/bundle.md` | Keep and verify | Describes implemented bundle parsing, discovery, validation, and digest semantics. |
|
||||
| `docs/internal/notify.md` | Keep and verify | Accurately states current no-op notification behavior. |
|
||||
| `docs/internal/publish.md` | Keep and verify | Describes planning, execution, replacement, safety, and current local scope. |
|
||||
| `docs/internal/state.md` | Keep and verify | Describes implemented `.distributor.json` state and comparison behavior. |
|
||||
| `docs/internal/storage.md` | Keep and verify | Describes storage interface, typed errors, path rules, traversal, and managed deletion. |
|
||||
| `docs/internal/transform.md` | Keep and verify | Describes Markdown-to-HTML sidecar behavior and transform boundaries. |
|
||||
| `docs/internal/app.md` | Added | Covers orchestration, backend factory, transform registry, dry-run, summaries, and notifier handoff. |
|
||||
| `docs/internal/config.md` | Added | Covers config loading, defaults, validation, accepted-but-not-executable backends, and example tests. |
|
||||
| `docs/internal/local-backend.md` | Missing, optional | Local adapter behavior may remain in `docs/internal/storage.md`; create this only if local filesystem safety detail outgrows that doc. |
|
||||
| `docs/integrations/markdown.md` | Added | Documents current Goldmark-backed Markdown rendering behavior. |
|
||||
| `docs/roadmap/audit.md` | Historical roadmap/report | Keep under roadmap unless replaced by a new audit. |
|
||||
| `docs/roadmap/cleanup.md` | Historical or completed roadmap | Keep under roadmap; optionally add completion status in the documentation refresh. |
|
||||
| `docs/roadmap/config.md` | Roadmap | Keep as planning material; avoid linking to it as current config reference. |
|
||||
| `docs/roadmap/contracts.md` | Roadmap | Keep as planning material; current implemented contracts should be summarized in `docs/internal/` and user docs as needed. |
|
||||
| `docs/roadmap/implementation.md` | Roadmap | Keep as implementation history plus future stages; status should be clear. |
|
||||
| `docs/roadmap/packages.md` | Roadmap | Keep as planning material; current package docs belong under `docs/internal/`. |
|
||||
| `docs/roadmap/storage.md` | Roadmap | Keep as planning material; current storage contract belongs in `docs/internal/storage.md`. |
|
||||
| `examples/local-to-local.yml` | Keep | Minimal local config; load-tested. |
|
||||
| `examples/local-publish.yml` | Keep | Runnable local publication example used by README and CLI docs. |
|
||||
| `examples/local-html.yml` | Keep | Runnable local HTML example. |
|
||||
| `examples/fan-out.yml` | Updated | Local-only runnable fan-out example. |
|
||||
| `examples/source-bundle/` | Keep | Copyable valid source bundle fixture for local CLI examples. |
|
||||
|
||||
Implementation source areas inspected for documentation truth:
|
||||
|
||||
- CLI entrypoints: `cmd/distributor`, `internal/cli`.
|
||||
- Application orchestration: `internal/app`.
|
||||
- Config loading/defaults/validation: `internal/config`.
|
||||
- Bundle manifest, discovery, and validation: `internal/bundle`.
|
||||
- Destination state: `internal/state`.
|
||||
- Storage abstraction and local/fake backends: `internal/storage`, `internal/storage/fake`, `internal/adapters/local`.
|
||||
- Publish planning and execution: `internal/publish`.
|
||||
- Transform registry and Markdown renderer: `internal/transform`, `internal/transform/markdown`.
|
||||
- Notification hook: `internal/notify`.
|
||||
- Tests and fixtures: package tests, `internal/testutil`, `examples/`, and `internal/bundle/testdata`.
|
||||
|
||||
Absent areas from earlier planning that should not be documented as implemented:
|
||||
|
||||
- `internal/adapters/ssh`
|
||||
- `internal/adapters/s3`
|
||||
- `internal/stage`
|
||||
- `internal/modules`
|
||||
- `internal/validators`
|
||||
- `internal/artifacts`
|
||||
- `internal/manifest`
|
||||
- `internal/schema`
|
||||
- `internal/report`
|
||||
- `pkg`
|
||||
|
||||
## Policy Compliance Assessment
|
||||
|
||||
Required current-behavior docs now exist for the config-driven, stateful, modular CLI.
|
||||
|
||||
Completed fixes from this refresh:
|
||||
|
||||
- `docs/policy/development.md` contains real workflow guidance.
|
||||
- Non-roadmap docs are scoped to implemented behavior. SSH/S3 execution, force overwrite, and external notification adapters remain described as unavailable unless the corresponding code exists.
|
||||
- `examples/fan-out.yml` is local-only and runnable.
|
||||
|
||||
Completed recommended fixes from this refresh:
|
||||
|
||||
- `docs/troubleshooting.md` covers recurring local MVP failure modes.
|
||||
- `docs/internal/app.md` and `docs/internal/config.md` provide current-behavior internal references.
|
||||
- `docs/integrations/markdown.md` documents current Goldmark-backed rendering behavior.
|
||||
- Historical roadmap status notes were added where useful.
|
||||
|
||||
No broad rewrite is needed for `README.md`, `docs/cli.md`, `docs/config.md`, or `docs/operations.md`. They are close to the implemented local MVP and should be tightened against code and tests.
|
||||
|
||||
## Target Documentation Set
|
||||
|
||||
### `README.md`
|
||||
|
||||
- Audience: users, administrators, operators.
|
||||
- Purpose: short project orientation and fastest useful local command.
|
||||
- Canonical scope: project purpose, elevator pitch, one local quickstart command, links to current docs.
|
||||
- Recommended outline: title, one-sentence description, local example command, links.
|
||||
- Source-of-truth repo areas to inspect: `internal/cli`, `internal/app/run.go`, `examples/local-publish.yml`, `docs/cli.md`.
|
||||
- Acceptance criteria: command is executable in the current local MVP; README does not describe SSH, S3, force overwrite, notification adapters, or future roadmap behavior as available.
|
||||
|
||||
### `docs/cli.md`
|
||||
|
||||
- Audience: users, administrators, operators.
|
||||
- Purpose: canonical CLI reference.
|
||||
- Canonical scope: commands, flags, useful workflows, command output expectations, local-only limits.
|
||||
- Recommended outline: shortest useful command, command overview, flag reference, common workflows, diagnostics and recovery commands.
|
||||
- Source-of-truth repo areas to inspect: `internal/cli/*.go`, `internal/cli/*_test.go`, `internal/app/validate.go`, `internal/app/inspect.go`, `internal/app/run.go`.
|
||||
- Acceptance criteria: every documented command and flag exists; `validate` and `inspect` are documented as local path commands; `run --dry-run` output is described without over-specifying every line; unsupported remote execution is stated clearly.
|
||||
|
||||
### `docs/config.md`
|
||||
|
||||
- Audience: administrators, operators, advanced users.
|
||||
- Purpose: canonical configuration reference.
|
||||
- Canonical scope: config file path behavior, minimal local config, production-oriented local config, full schema, defaults, validation rules, secrets handling, links to examples.
|
||||
- Recommended outline: config file location, minimal local config, production-oriented local config, reference, defaults, secrets, examples.
|
||||
- Source-of-truth repo areas to inspect: `internal/config/config.go`, `internal/config/defaults.go`, `internal/config/load.go`, `internal/config/validate.go`, `internal/config/load_test.go`, `examples/*.yml`.
|
||||
- Acceptance criteria: fields and defaults match code; `KnownFields(true)` behavior is noted where useful; SSH/S3 fields are described as accepted by config validation but not implemented for execution; `on_digest_mismatch: warn` and unmanaged overwrite are not documented as active options.
|
||||
|
||||
### `docs/operations.md`
|
||||
|
||||
- Audience: administrators, operators.
|
||||
- Purpose: operating and recovery notes for the implemented local MVP.
|
||||
- Canonical scope: local workflow, filesystem layout, destination state, dry-run, retry behavior, replacement safety, failed write cleanup, caveats.
|
||||
- Recommended outline: normal workflow, filesystem layout, destination state, dry-run and planning, retry and replacement behavior, failure handling, cleanup behavior, caveats.
|
||||
- Source-of-truth repo areas to inspect: `internal/app/run.go`, `internal/publish/plan.go`, `internal/publish/execute.go`, `internal/publish/reconcile.go`, `internal/publish/safety.go`, `internal/state`, `internal/adapters/local`.
|
||||
- Acceptance criteria: describes only local-to-local operation; explains `.distributor.json` as the managed sentinel; distinguishes skip, replace, conflict, and unmanaged destination behavior; does not promise resume, remote storage, force overwrite, or external notifications.
|
||||
|
||||
### `docs/troubleshooting.md`
|
||||
|
||||
- Audience: administrators, operators.
|
||||
- Purpose: symptom-oriented fixes for common local MVP failures.
|
||||
- Canonical scope: implemented failure modes only.
|
||||
- Recommended outline: one entry per symptom with symptom, likely cause, diagnostic step, safe fix, and relevant link.
|
||||
- Source-of-truth repo areas to inspect: `internal/config/validate.go`, `internal/bundle/validate.go`, `internal/state/compare.go`, `internal/publish/plan.go`, `internal/publish/output.go`, CLI tests.
|
||||
- Acceptance criteria: entries are actionable and do not suggest unsafe deletion; remote backend failures are described only as unsupported execution; all fixes link to `docs/cli.md`, `docs/config.md`, or `docs/operations.md` where useful.
|
||||
|
||||
### `docs/policy/architecture.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: development principles and architectural invariants.
|
||||
- Canonical scope: project shape, package boundaries, state/persistence philosophy, external integration philosophy, errors/logging, tests, docs, non-goals.
|
||||
- Recommended outline: keep the existing outline.
|
||||
- Source-of-truth repo areas to inspect: full package tree, implemented internal docs, roadmap files for explicitly future work.
|
||||
- Acceptance criteria: still gives long-term architecture direction, but any unimplemented adapter packages or future capabilities are worded as planned/target architecture rather than implemented behavior.
|
||||
|
||||
### `docs/policy/development.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: contributor and agent workflow.
|
||||
- Canonical scope: repo layout, build/test commands, coding conventions, dependency policy, how to add config fields, CLI flags, backends, transforms, examples, and docs.
|
||||
- Recommended outline: repository layout, common commands, coding conventions, dependency policy, adding config fields, adding CLI flags, adding storage backends, adding transforms, updating examples, documentation expectations.
|
||||
- Source-of-truth repo areas to inspect: `go.mod`, `cmd/distributor`, `internal/*`, `examples`, tests, `docs/policy/architecture.md`, `docs/policy/documentation.md`.
|
||||
- Acceptance criteria: no placeholder content remains; commands are real; workflow guidance protects current boundaries; examples and docs update rules match policy.
|
||||
|
||||
### `docs/internal/app.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: implemented orchestration reference.
|
||||
- Canonical scope: `Run`, `Validate`, `Inspect`, backend factory, transform registry, dry-run, per-destination fan-out, failure aggregation, notifier invocation.
|
||||
- Recommended outline: purpose, inputs and outputs, run flow, backend and transform registration, dry-run behavior, failure behavior, notification behavior, tests to inspect, invariants.
|
||||
- Source-of-truth repo areas to inspect: `internal/app/*.go`, `internal/app/*_test.go`, `internal/cli/root_test.go`.
|
||||
- Acceptance criteria: documents current local-only backend execution and the no-op default notifier; does not introduce a generic stage framework that does not exist.
|
||||
|
||||
### `docs/internal/config.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: internal config loading/default/validation reference.
|
||||
- Canonical scope: YAML decoding, known-field rejection, defaults, validation error model, backend config shape, publish/transform validation helper, example load tests.
|
||||
- Recommended outline: purpose, inputs and outputs, loading flow, defaults, validation responsibilities, executable support boundary, tests to inspect, invariants.
|
||||
- Source-of-truth repo areas to inspect: `internal/config/*.go`, `internal/config/*_test.go`, `docs/config.md`, `examples/*.yml`.
|
||||
- Acceptance criteria: documents that SSH/S3 config validation exists while execution does not; keeps user-facing config reference canonical in `docs/config.md`.
|
||||
|
||||
### `docs/internal/bundle.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: implemented bundle contract and validation reference.
|
||||
- Canonical scope: `manifest.json`, discovery, validation, digest semantics, storage interactions, tests.
|
||||
- Recommended outline: keep current outline and verify against code.
|
||||
- Source-of-truth repo areas to inspect: `internal/bundle`, `internal/storage`, `internal/bundle/testdata`, `examples/source-bundle`.
|
||||
- Acceptance criteria: canonical digest, duplicate paths, reserved paths, symlink rejection, RFC3339 parsing, and discovery behavior match implementation.
|
||||
|
||||
### `docs/internal/state.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: destination state and comparison reference.
|
||||
- Canonical scope: `.distributor.json` schema, validation, output metadata, comparison outcomes.
|
||||
- Recommended outline: keep current outline and verify against code.
|
||||
- Source-of-truth repo areas to inspect: `internal/state`, `internal/publish/reconcile.go`, `internal/publish/execute.go`.
|
||||
- Acceptance criteria: state schema and comparison outcomes match implemented structs and tests; `distributor_version` is described as optional diagnostic metadata.
|
||||
|
||||
### `docs/internal/storage.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: storage interface and backend safety reference.
|
||||
- Canonical scope: logical paths, IO methods, traversal, `HasAny`, typed errors, managed deletion, local and fake backend behavior.
|
||||
- Recommended outline: keep current outline and add any missing implemented details that matter for callers.
|
||||
- Source-of-truth repo areas to inspect: `internal/storage`, `internal/storage/fake`, `internal/adapters/local`.
|
||||
- Acceptance criteria: matches actual `Backend` interface, `WriteOptions`, `DeleteOptions`, `ErrStopWalk`, and `storage.List` helper; does not describe raw recursive delete as available.
|
||||
|
||||
### `docs/internal/publish.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: publish planning and execution reference.
|
||||
- Canonical scope: request inputs, output planning, destination inspection, transfer policy, actions, replacement safety, cleanup on failed writes.
|
||||
- Recommended outline: keep current outline and verify against code.
|
||||
- Source-of-truth repo areas to inspect: `internal/publish`, `internal/app/run.go`, `internal/config/defaults.go`.
|
||||
- Acceptance criteria: action names match constants; transfer policy values match code; collision detection and managed deletion behavior are covered.
|
||||
|
||||
### `docs/internal/transform.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: transform registry and Markdown transform reference.
|
||||
- Canonical scope: transform interface, registry, Markdown sidecar output, deterministic output metadata, raw HTML behavior.
|
||||
- Recommended outline: keep current outline; link to integration notes if `docs/integrations/markdown.md` is created.
|
||||
- Source-of-truth repo areas to inspect: `internal/transform`, `internal/transform/markdown`, markdown tests.
|
||||
- Acceptance criteria: `.md` to `.html` sidecar naming, skipped non-Markdown files, digest metadata, and source immutability match implementation.
|
||||
|
||||
### `docs/internal/notify.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: notification hook reference.
|
||||
- Canonical scope: interface, no-op notifier, invocation points, non-invocation points.
|
||||
- Recommended outline: keep current outline and add tests to inspect if useful.
|
||||
- Source-of-truth repo areas to inspect: `internal/notify`, `internal/app/run.go`, `internal/app/run_test.go`.
|
||||
- Acceptance criteria: says no external notification adapters or user-facing notification config exist.
|
||||
|
||||
### `docs/integrations/markdown.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: concise external integration note for Markdown rendering.
|
||||
- Canonical scope: Goldmark dependency, renderer defaults used by `markdown.New`, raw HTML behavior as observed in tests, deterministic wrapper template, supported output mode.
|
||||
- Recommended outline: purpose, dependency, behavior used, behavior intentionally not customized, tests to inspect, update rules.
|
||||
- Source-of-truth repo areas to inspect: `go.mod`, `internal/transform/markdown`, markdown tests.
|
||||
- Acceptance criteria: documents only the Markdown renderer behavior actually used; does not claim full CommonMark compatibility beyond Goldmark defaults.
|
||||
|
||||
### `docs/roadmap/*.md`
|
||||
|
||||
- Audience: maintainers, developers, LLM coding agents.
|
||||
- Purpose: future work, historical plans, accepted deferred work, and implementation prompts.
|
||||
- Canonical scope: unimplemented SSH/S3 adapters, force overwrite, notification adapters, future config fields, release readiness, historical cleanup/audit plans.
|
||||
- Recommended outline: add status notes only where useful; avoid rewriting history unless it causes confusion.
|
||||
- Source-of-truth repo areas to inspect: current implementation and each roadmap file.
|
||||
- Acceptance criteria: future work remains under `docs/roadmap/`; completed historical plans are labeled clearly enough that agents do not re-run them blindly.
|
||||
|
||||
## File-by-File Rewrite Guidance
|
||||
|
||||
`README.md`:
|
||||
|
||||
- Cover: purpose, one local quickstart, links.
|
||||
- Avoid: full config schema, internal package details, remote backend promises.
|
||||
- Link to: `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/roadmap/`.
|
||||
- Inspect: `examples/local-publish.yml`, `internal/cli`.
|
||||
- Stale claims to remove: any implication that remote publication is implemented.
|
||||
|
||||
`docs/cli.md`:
|
||||
|
||||
- Cover: real command syntax and current local workflows.
|
||||
- Avoid: roadmap flags such as force overwrite or remote validation.
|
||||
- Link to: `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md` if created.
|
||||
- Inspect: `internal/cli/*_test.go`.
|
||||
- Stale claims to remove: any command or flag not present in `internal/cli`.
|
||||
|
||||
`docs/config.md`:
|
||||
|
||||
- Cover: current config schema and defaults.
|
||||
- Avoid: presenting SSH/S3 as executable backend support.
|
||||
- Link to: examples and operations.
|
||||
- Inspect: `internal/config/defaults.go`, `internal/config/validate.go`, `internal/config/load_test.go`.
|
||||
- Stale claims to remove: `on_digest_mismatch: warn`, unmanaged overwrite, or force replacement as active options.
|
||||
|
||||
`docs/operations.md`:
|
||||
|
||||
- Cover: local destination state, managed cleanup, retry behavior, fan-out failure aggregation.
|
||||
- Avoid: remote storage recovery, force cleanup, external notifier delivery.
|
||||
- Link to: `docs/troubleshooting.md` for symptom-specific fixes.
|
||||
- Inspect: `internal/app/run.go`, `internal/publish/execute.go`, `internal/state`.
|
||||
- Stale claims to remove: any resume or recovery mechanism beyond re-running after safe cleanup.
|
||||
|
||||
`docs/policy/development.md`:
|
||||
|
||||
- Cover: concrete contributor workflow.
|
||||
- Avoid: placeholder text and invented tools.
|
||||
- Link to: architecture and documentation policies.
|
||||
- Inspect: package tree and `go.mod`.
|
||||
- Stale claims to remove: `# Not yet implemented`.
|
||||
|
||||
`docs/internal/*.md`:
|
||||
|
||||
- Cover: implemented component contracts, boundaries, failure behavior, tests to inspect.
|
||||
- Avoid: roadmap package names or future stages as if they exist.
|
||||
- Link to: current user docs only when relevant.
|
||||
- Inspect: package code and tests.
|
||||
- Stale claims to remove: broad future backend behavior outside local/fake abstractions.
|
||||
|
||||
`examples/`:
|
||||
|
||||
- Cover: valid, maintained, copyable examples.
|
||||
- Avoid: examples that look runnable but fail because the backend execution is unsupported.
|
||||
- Link from: README, CLI docs, config docs.
|
||||
- Inspect: `internal/config/load_test.go` and optional CLI smoke commands.
|
||||
- Stale claims to remove: executable remote fan-out examples until remote backends exist.
|
||||
|
||||
## Examples Plan
|
||||
|
||||
Keep these examples:
|
||||
|
||||
- `examples/source-bundle/`: valid source bundle used by local CLI examples.
|
||||
- `examples/local-to-local.yml`: minimal local config. Use in config docs as the minimal schema example.
|
||||
- `examples/local-publish.yml`: primary runnable quickstart config.
|
||||
- `examples/local-html.yml`: runnable Markdown-to-HTML example.
|
||||
|
||||
Revise `examples/fan-out.yml` in the documentation refresh:
|
||||
|
||||
- Preferred option: replace it with a local-only fan-out example using two local destinations, such as one source archive destination and one HTML destination under `workspace/`.
|
||||
- Alternative option: move the SSH/S3 fan-out material under a roadmap file and remove it from `examples/`.
|
||||
- Do not keep a non-roadmap example that appears copyable for execution but uses unsupported SSH/S3 execution.
|
||||
|
||||
Future example tests should continue loading every YAML file under `examples/`. If `examples/fan-out.yml` becomes local-only, add or update a CLI/app test that exercises local fan-out behavior or rely on existing fan-out tests if they cover equivalent behavior.
|
||||
|
||||
## Internal Documentation Plan
|
||||
|
||||
Update existing internal docs only after checking package code and tests.
|
||||
|
||||
Create these internal docs:
|
||||
|
||||
- `docs/internal/app.md`: orchestration, backend factory, transform registry, dry-run behavior, fan-out, failure aggregation, notifier invocation.
|
||||
- `docs/internal/config.md`: YAML loading, defaults, validation, accepted config fields, unsupported execution boundary, example load tests.
|
||||
|
||||
Keep and verify these internal docs:
|
||||
|
||||
- `docs/internal/bundle.md`
|
||||
- `docs/internal/state.md`
|
||||
- `docs/internal/storage.md`
|
||||
- `docs/internal/publish.md`
|
||||
- `docs/internal/transform.md`
|
||||
- `docs/internal/notify.md`
|
||||
|
||||
Defer these internal docs unless the corresponding implementation grows:
|
||||
|
||||
- `docs/internal/local-backend.md`: create only if local adapter details become too long for `docs/internal/storage.md`.
|
||||
- `docs/internal/logging.md`: defer until logging has behavior beyond placeholders.
|
||||
- `docs/internal/testutil.md`: defer unless test fixture helpers become a stable contributor-facing contract.
|
||||
|
||||
Do not create docs for packages or directories that do not exist.
|
||||
|
||||
## Integration Documentation Plan
|
||||
|
||||
No external service integration docs should be created for SSH, S3, or notification services until those integrations are implemented.
|
||||
|
||||
Recommended current integration doc:
|
||||
|
||||
- `docs/integrations/markdown.md`
|
||||
|
||||
This should document Goldmark usage because Markdown rendering is an implemented external file-format integration with externally visible output. Keep it concise and limited to:
|
||||
|
||||
- dependency and version source: `go.mod`;
|
||||
- renderer construction: `goldmark.New()`;
|
||||
- sidecar output path behavior owned by `internal/transform/markdown`;
|
||||
- raw HTML behavior covered by tests;
|
||||
- wrapper template behavior;
|
||||
- tests to inspect before changing renderer behavior.
|
||||
|
||||
Do not create separate YAML integration docs unless configuration parsing behavior outgrows `docs/config.md` and `docs/internal/config.md`.
|
||||
|
||||
## Recommended Implementation Sequence
|
||||
|
||||
### Stage 1: Establish Documentation Status and Contributor Policy
|
||||
|
||||
- Goal: remove the required policy placeholder and classify active versus historical documentation.
|
||||
- Files to create/update/delete/move: update `docs/policy/development.md`; optionally add short status notes to `docs/roadmap/audit.md` and `docs/roadmap/cleanup.md`.
|
||||
- Repo areas to inspect: `go.mod`, `cmd/distributor`, `internal/*`, `examples`, `docs/policy/architecture.md`, `docs/policy/documentation.md`.
|
||||
- Acceptance criteria: `docs/policy/development.md` contains real workflow guidance; no active policy doc says "Not yet implemented"; roadmap status notes do not change current behavior docs.
|
||||
- Suggested validation commands: `rg -n "Not yet implemented" docs/policy README.md docs/*.md docs/internal`; `git diff -- docs/policy/development.md docs/roadmap/audit.md docs/roadmap/cleanup.md`.
|
||||
- One prompt: yes.
|
||||
|
||||
### Stage 2: Tighten Current User and Operator Docs
|
||||
|
||||
- Goal: make README, CLI, config, and operations docs exactly match the local MVP.
|
||||
- Files to create/update/delete/move: update `README.md`, `docs/cli.md`, `docs/config.md`, `docs/operations.md`; create `docs/troubleshooting.md`.
|
||||
- Repo areas to inspect: `internal/cli`, `internal/app`, `internal/config`, `internal/publish`, `internal/state`, `examples`.
|
||||
- Acceptance criteria: user docs describe local execution, local validation/inspection, current config schema, current defaults, current state/retry behavior, and clear unsupported remote execution boundaries.
|
||||
- Suggested validation commands: `rg -n "force|allow_unmanaged|on_digest_mismatch: warn|warn" README.md docs/cli.md docs/config.md docs/operations.md docs/troubleshooting.md`; `rg -n "ssh|s3|remote|notification" README.md docs/cli.md docs/config.md docs/operations.md docs/troubleshooting.md`.
|
||||
- One prompt: yes.
|
||||
|
||||
### Stage 3: Make Examples Fully Runnable or Clearly Roadmap-Only
|
||||
|
||||
- Goal: ensure `examples/` contains implemented, copyable examples only.
|
||||
- Files to create/update/delete/move: update or replace `examples/fan-out.yml`; update links in `README.md`, `docs/cli.md`, and `docs/config.md` if needed; move remote fan-out material to a roadmap section if preserving it is useful.
|
||||
- Repo areas to inspect: `internal/config/load_test.go`, `internal/app/run_test.go`, `examples`.
|
||||
- Acceptance criteria: every example under `examples/` is valid current config and does not rely on unsupported remote execution; primary examples remain load-tested.
|
||||
- Suggested validation commands: `go test ./internal/config`; optional `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`; optional `go run ./cmd/distributor run --config examples/local-html.yml --dry-run`.
|
||||
- One prompt: yes.
|
||||
|
||||
### Stage 4: Complete Internal Component Docs
|
||||
|
||||
- Goal: give future agents a current-behavior internal map before remote backends are added.
|
||||
- Files to create/update/delete/move: create `docs/internal/app.md` and `docs/internal/config.md`; update existing `docs/internal/*.md` as needed.
|
||||
- Repo areas to inspect: `internal/app`, `internal/config`, `internal/bundle`, `internal/state`, `internal/storage`, `internal/storage/fake`, `internal/adapters/local`, `internal/publish`, `internal/transform`, `internal/notify`, package tests.
|
||||
- Acceptance criteria: every major implemented component has a concise doc with purpose, inputs/outputs, boundaries, failure behavior, tests to inspect, and invariants; no internal doc describes absent SSH/S3 adapters as implemented.
|
||||
- Suggested validation commands: `rg -n "internal/adapters/ssh|internal/adapters/s3|not implemented|future" docs/internal`; `git diff -- docs/internal`.
|
||||
- One prompt: yes.
|
||||
|
||||
### Stage 5: Add Markdown Integration Notes
|
||||
|
||||
- Goal: document the one implemented external file-format integration where behavior matters.
|
||||
- Files to create/update/delete/move: create `docs/integrations/markdown.md`; optionally link from `docs/internal/transform.md`.
|
||||
- Repo areas to inspect: `go.mod`, `internal/transform/markdown`, markdown tests.
|
||||
- Acceptance criteria: the doc is concise, version-aware through `go.mod`, and limited to current Goldmark usage and observed renderer behavior.
|
||||
- Suggested validation commands: `go test ./internal/transform/markdown`; `rg -n "Goldmark|markdown" docs/integrations docs/internal/transform.md`.
|
||||
- One prompt: yes.
|
||||
|
||||
### Stage 6: Final Documentation Consistency Sweep
|
||||
|
||||
- Goal: catch stale links, stale roadmap references, and unimplemented claims outside roadmap.
|
||||
- Files to create/update/delete/move: any docs touched in earlier stages.
|
||||
- Repo areas to inspect: all docs and examples.
|
||||
- Acceptance criteria: documentation is current, concise, link-consistent, and policy-compliant.
|
||||
- Suggested validation commands: `go test ./...`; `rg -n "go-application-template|maximumdirect.net|docs/architecture.md|docs/documentation.md" README.md docs examples`; `rg -n "allow_unmanaged_overwrite|on_digest_mismatch: warn" README.md docs examples`; `rg -n "not implemented|future|planned|roadmap|ssh|s3|remote" README.md docs/*.md docs/internal docs/policy examples`.
|
||||
- One prompt: yes, after the prior stages are complete.
|
||||
|
||||
## Validation Plan
|
||||
|
||||
Run Go tests when examples, CLI docs, config docs, or behavior-linked docs change:
|
||||
|
||||
- `go test ./internal/config`: validates example config loading and config semantics.
|
||||
- `go test ./internal/cli ./internal/app`: validates documented command behavior and local run workflows.
|
||||
- `go test ./internal/transform/markdown`: validates documented Markdown rendering behavior.
|
||||
- `go test ./...`: final broad verification after documentation and example changes.
|
||||
|
||||
Run grep checks:
|
||||
|
||||
- `rg -n "Not yet implemented" docs/policy README.md docs/*.md docs/internal`
|
||||
- `rg -n "go-application-template|docs/architecture.md|docs/documentation.md" README.md docs examples`
|
||||
- `rg -n "allow_unmanaged_overwrite|on_digest_mismatch: warn" README.md docs examples`
|
||||
- `rg -n "force overwrite|--force|remote backends are implemented|notification adapters" README.md docs/*.md docs/internal docs/policy examples`
|
||||
- `rg -n "ssh|s3|remote" README.md docs/*.md docs/internal docs/policy examples`
|
||||
|
||||
The final SSH/S3/remote grep is not expected to return zero results. Manually review every result and confirm it is either:
|
||||
|
||||
- under `docs/roadmap/`;
|
||||
- a clearly stated unsupported-execution boundary;
|
||||
- a config-validation reference that does not imply executable support; or
|
||||
- an architecture policy statement phrased as future/target direction rather than implemented behavior.
|
||||
|
||||
Manual review checklist:
|
||||
|
||||
- README remains short.
|
||||
- `docs/config.md` is the only current-behavior config reference.
|
||||
- `docs/cli.md` is the only current-behavior CLI reference.
|
||||
- `docs/operations.md` covers state and recovery without unsafe deletion advice.
|
||||
- `docs/troubleshooting.md` is symptom-oriented and links to canonical docs.
|
||||
- Internal docs point to tests before changing behavior.
|
||||
- Examples are copyable and free of secrets.
|
||||
- Future work remains under `docs/roadmap/`.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No blocking questions remain before implementing this documentation refresh.
|
||||
|
||||
The only non-blocking choice is how to handle `examples/fan-out.yml`:
|
||||
|
||||
- Preferred: convert it to a local-only fan-out example so `examples/` remains fully runnable.
|
||||
- Acceptable: move the current SSH/S3 fan-out example into roadmap material until remote execution exists.
|
||||
|
||||
Use the preferred option unless a maintainer explicitly wants `examples/` to include config-validated but non-executable examples.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,526 +0,0 @@
|
||||
# Package Layout Roadmap
|
||||
|
||||
This roadmap defines the proposed package layout, boundaries, and implementation responsibilities for the `distributor` MVP.
|
||||
|
||||
`distributor` is expected to be a domain-agnostic bundle publisher. Producer applications emit source bundles containing `manifest.json`; `distributor` validates those bundles and publishes selected source and generated artifacts to one or more configured destinations.
|
||||
|
||||
This document is roadmap material. It describes the intended package design before implementation and should move into `docs/internal/` only after corresponding behavior exists.
|
||||
|
||||
## Accepted Package Layout
|
||||
|
||||
```text
|
||||
cmd/distributor/
|
||||
main.go
|
||||
|
||||
internal/app/
|
||||
app.go
|
||||
run.go
|
||||
pipeline.go
|
||||
|
||||
internal/cli/
|
||||
root.go
|
||||
run.go
|
||||
validate.go
|
||||
inspect.go
|
||||
|
||||
internal/config/
|
||||
config.go
|
||||
defaults.go
|
||||
load.go
|
||||
validate.go
|
||||
|
||||
internal/bundle/
|
||||
manifest.go
|
||||
digest.go
|
||||
validate.go
|
||||
discover.go
|
||||
|
||||
internal/state/
|
||||
distributor.go
|
||||
compare.go
|
||||
validate.go
|
||||
|
||||
internal/storage/
|
||||
backend.go
|
||||
registry.go
|
||||
path.go
|
||||
errors.go
|
||||
|
||||
internal/storage/fake/
|
||||
backend.go
|
||||
|
||||
internal/adapters/local/
|
||||
backend.go
|
||||
|
||||
internal/adapters/ssh/
|
||||
backend.go
|
||||
config.go
|
||||
|
||||
internal/adapters/s3/
|
||||
backend.go
|
||||
config.go
|
||||
|
||||
internal/transform/
|
||||
transform.go
|
||||
registry.go
|
||||
plan.go
|
||||
|
||||
internal/transform/markdown/
|
||||
markdown.go
|
||||
template.go
|
||||
|
||||
internal/publish/
|
||||
plan.go
|
||||
execute.go
|
||||
output.go
|
||||
reconcile.go
|
||||
safety.go
|
||||
|
||||
internal/notify/
|
||||
notify.go
|
||||
noop.go
|
||||
|
||||
internal/logging/
|
||||
logging.go
|
||||
```
|
||||
|
||||
## Package Responsibilities
|
||||
|
||||
### `cmd/distributor`
|
||||
|
||||
Application entrypoint only.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- call CLI execution;
|
||||
- translate process exit status;
|
||||
- avoid business logic.
|
||||
|
||||
Non-responsibilities:
|
||||
|
||||
- config loading;
|
||||
- backend construction;
|
||||
- bundle validation;
|
||||
- publish decisions.
|
||||
|
||||
### `internal/cli`
|
||||
|
||||
CLI command definitions, flags, argument parsing, and command wiring.
|
||||
|
||||
Expected MVP commands:
|
||||
|
||||
- `distributor run` — run configured pipelines.
|
||||
- `distributor run --dry-run` — plan without modifying destinations.
|
||||
- `distributor run --pipeline <id>` — run one configured pipeline.
|
||||
- `distributor validate <path>` — validate a source bundle or source tree where feasible.
|
||||
- `distributor inspect <path>` — inspect a bundle or destination state where feasible.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- CLI should call `internal/app` use cases.
|
||||
- CLI should not parse manifests directly except through application APIs.
|
||||
- CLI should not import backend adapter implementation details unless only for registration side effects.
|
||||
|
||||
### `internal/app`
|
||||
|
||||
Application orchestration and top-level use cases.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- load and validate configuration;
|
||||
- construct configured pipelines;
|
||||
- build source and destination backends through registries;
|
||||
- orchestrate discovery, validation, planning, publishing, and notification;
|
||||
- coordinate dry-run output;
|
||||
- run destination fan-out deterministically and sequentially;
|
||||
- aggregate destination outcomes into run-level failure behavior.
|
||||
|
||||
Core orchestration shape:
|
||||
|
||||
```text
|
||||
for each selected pipeline:
|
||||
open source backend
|
||||
discover source bundles
|
||||
for each source bundle:
|
||||
validate source manifest and digest
|
||||
for each destination:
|
||||
inspect .distributor.json
|
||||
build publish plan
|
||||
transform as required by that destination
|
||||
execute publish plan unless dry-run
|
||||
run noop notifier after actual publication or replacement
|
||||
```
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/app` composes packages but should not contain backend-specific logic.
|
||||
- Publish decisions should live in `internal/publish`, not inline in orchestration.
|
||||
- Destination state comparison should live in `internal/state` or `internal/publish`, not CLI code.
|
||||
|
||||
### `internal/config`
|
||||
|
||||
Configuration structs, defaults, loading, precedence, and validation.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- load `/usr/local/etc/distributor/config.yml` by default;
|
||||
- support `--config` override;
|
||||
- apply defaults;
|
||||
- validate required fields;
|
||||
- validate pipeline ids and destination ids;
|
||||
- validate backend-specific config shapes;
|
||||
- validate transform and publish policy combinations.
|
||||
|
||||
MVP config model:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
Configuration principles:
|
||||
|
||||
- one source per pipeline;
|
||||
- one or more destinations per pipeline;
|
||||
- transforms are destination-specific;
|
||||
- publish policy is destination-specific;
|
||||
- secrets should use environment variables, secret files, SSH agent, or standard credential mechanisms rather than raw YAML values.
|
||||
|
||||
### `internal/bundle`
|
||||
|
||||
Source bundle contract and validation.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- parse source `manifest.json`;
|
||||
- represent source manifests and files;
|
||||
- discover bundle roots beneath a configured source root;
|
||||
- validate required manifest fields;
|
||||
- validate RFC3339 `created` values;
|
||||
- validate relative paths;
|
||||
- validate file existence, size, per-file SHA-256, and bundle digest;
|
||||
- expose normalized source bundle models to other packages.
|
||||
|
||||
Core types:
|
||||
|
||||
```go
|
||||
type Manifest struct {
|
||||
SchemaVersion int
|
||||
ID string
|
||||
Digest string
|
||||
Created time.Time
|
||||
Files []ManifestFile
|
||||
}
|
||||
|
||||
type ManifestFile struct {
|
||||
Path string
|
||||
SHA256 string
|
||||
Size int64
|
||||
}
|
||||
|
||||
type Bundle struct {
|
||||
RootRelativePath string
|
||||
Manifest Manifest
|
||||
}
|
||||
```
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/bundle` does not know about `.distributor.json`.
|
||||
- `internal/bundle` does not know about destinations, transforms, or notification.
|
||||
- `internal/bundle` may use the storage abstraction to read source files, but it should not import backend adapter packages.
|
||||
|
||||
### `internal/state`
|
||||
|
||||
Destination state contract for `.distributor.json`.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- parse `.distributor.json`;
|
||||
- validate destination state;
|
||||
- represent copied source outputs and generated outputs;
|
||||
- embed the source manifest used for publication;
|
||||
- compare destination state against a current source manifest;
|
||||
- classify destination state as same, older, newer, conflict, absent, invalid, or unmanaged.
|
||||
|
||||
Core types:
|
||||
|
||||
```go
|
||||
type DistributorState struct {
|
||||
SchemaVersion int
|
||||
DistributorVersion string
|
||||
PipelineID string
|
||||
DestinationID string
|
||||
PublishedAt time.Time
|
||||
Source SourceState
|
||||
Outputs []OutputFile
|
||||
}
|
||||
|
||||
type SourceState struct {
|
||||
Manifest bundle.Manifest
|
||||
}
|
||||
|
||||
type OutputFile struct {
|
||||
Path string
|
||||
Kind string // source | generated
|
||||
SourcePath string
|
||||
Transform string
|
||||
SHA256 string
|
||||
Size int64
|
||||
}
|
||||
```
|
||||
|
||||
Comparison rules:
|
||||
|
||||
- same source manifest: skip;
|
||||
- same source id, older destination source `created`: replace;
|
||||
- same source id, newer destination source `created`: skip;
|
||||
- same source id, same `created`, different digest: conflict;
|
||||
- different source id: conflict;
|
||||
- pipeline id or destination id mismatch: conflict;
|
||||
- absent state: publish only if safe;
|
||||
- unmanaged non-empty path: fail.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/state` owns destination state semantics, not publish execution.
|
||||
- `internal/state` should not know about S3, SSH/SFTP, local filesystem details, or Markdown rendering.
|
||||
|
||||
### `internal/storage`
|
||||
|
||||
Backend abstraction and shared storage types.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- define storage backend interfaces;
|
||||
- define object/file metadata types;
|
||||
- define path/prefix helpers;
|
||||
- define common storage errors;
|
||||
- provide backend registry mechanisms;
|
||||
- provide a fake backend for core package tests.
|
||||
|
||||
The detailed storage contract is defined in `docs/roadmap/storage.md`. Core application code should use that storage interface for backend-rooted logical paths, byte and stream IO, metadata, traversal, typed errors, emptiness checks, and managed deletion.
|
||||
|
||||
Destructive APIs should remain narrow. Prefer managed deletion of files recorded in `.distributor.json` instead of broad recursive deletion.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/storage` should not contain backend implementation details.
|
||||
- Adapter dependencies must not leak through storage interfaces.
|
||||
- The fake backend exists for tests and should not become an application runtime backend.
|
||||
|
||||
### `internal/adapters/local`
|
||||
|
||||
Local filesystem backend.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- implement `storage.Backend` for local paths;
|
||||
- clean and constrain paths;
|
||||
- perform safe reads/writes/listing/deletion;
|
||||
- use atomic writes where practical;
|
||||
- reject unsafe path traversal;
|
||||
- handle symlink policy explicitly.
|
||||
|
||||
Testing expectations:
|
||||
|
||||
- use temporary directories;
|
||||
- verify path traversal rejection;
|
||||
- verify write and delete safety.
|
||||
|
||||
### `internal/adapters/ssh`
|
||||
|
||||
SSH/SFTP backend.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- implement `storage.Backend` over SSH/SFTP;
|
||||
- support `uri` and `path` config;
|
||||
- prefer native SFTP implementation;
|
||||
- use SSH agent, key files, known hosts, or documented auth mechanisms;
|
||||
- avoid raw passwords in config unless explicitly designed and documented later;
|
||||
- translate SSH/SFTP errors into storage-level errors.
|
||||
|
||||
Testing expectations:
|
||||
|
||||
- core app tests should use fake backends;
|
||||
- adapter tests may use local test servers or targeted integration tests if practical;
|
||||
- do not require a real production SSH host for normal unit tests.
|
||||
|
||||
### `internal/adapters/s3`
|
||||
|
||||
S3-compatible object storage backend.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- implement `storage.Backend` over S3-compatible object storage;
|
||||
- support endpoint, bucket, prefix, region, and force-path-style configuration;
|
||||
- support standard credential mechanisms or explicit environment-variable references;
|
||||
- treat S3 as an object tree, not a filesystem;
|
||||
- set reasonable content types where practical;
|
||||
- guard against prefix/root deletion mistakes.
|
||||
|
||||
Testing expectations:
|
||||
|
||||
- core app tests should use fake backends;
|
||||
- adapter behavior may be tested through mocks, local S3-compatible services, or narrow integration tests;
|
||||
- config examples should avoid real secrets.
|
||||
|
||||
### `internal/transform`
|
||||
|
||||
Transform interfaces, registry, and transform planning.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- define transform interfaces;
|
||||
- register available transforms;
|
||||
- represent transform requests and outputs;
|
||||
- keep transform execution independent of destination backend details.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- transforms operate on source bundle content and destination transform config;
|
||||
- transforms do not publish outputs;
|
||||
- transforms do not mutate source bundles;
|
||||
- transforms should return generated output metadata for `.distributor.json`.
|
||||
|
||||
### `internal/transform/markdown`
|
||||
|
||||
Markdown-to-HTML implementation.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- render listed Markdown files to HTML;
|
||||
- support MVP sidecar behavior, such as `report.md` -> `report.html`;
|
||||
- record generated output path, source path, transform name, SHA-256, and size;
|
||||
- optionally use embedded templates if needed.
|
||||
|
||||
MVP scope:
|
||||
|
||||
- Markdown to HTML only;
|
||||
- no PDF generation;
|
||||
- no email-specific HTML;
|
||||
- no complex theming unless required for basic output correctness.
|
||||
|
||||
### `internal/publish`
|
||||
|
||||
Destination planning, reconciliation, safety checks, and publish execution.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- inspect destination state;
|
||||
- plan destination action;
|
||||
- enforce destination conflict rules;
|
||||
- enforce destructive-operation safety rules;
|
||||
- detect output path collisions before writing;
|
||||
- combine source files and transform outputs according to destination publish policy;
|
||||
- write destination outputs;
|
||||
- write `.distributor.json`;
|
||||
- use staging or equivalent cleanup behavior where practical;
|
||||
- support dry-run planning;
|
||||
- report skipped, replaced, failed, and published actions.
|
||||
|
||||
Action model:
|
||||
|
||||
```text
|
||||
publish
|
||||
replace
|
||||
skip_same
|
||||
skip_destination_newer
|
||||
fail_conflict
|
||||
fail_unmanaged
|
||||
```
|
||||
|
||||
Boundaries:
|
||||
|
||||
- publish logic should not parse CLI flags;
|
||||
- publish logic should not know adapter implementation details;
|
||||
- publish logic should use `internal/state` for destination state semantics;
|
||||
- publish logic should use `internal/storage` interfaces for IO.
|
||||
|
||||
### `internal/notify`
|
||||
|
||||
Notification stage abstraction.
|
||||
|
||||
MVP responsibilities:
|
||||
|
||||
- define notifier interface;
|
||||
- implement no-op notifier;
|
||||
- preserve future extension point for email, ntfy, Gotify, RSS update hooks, or other notification channels.
|
||||
|
||||
Future notification rules:
|
||||
|
||||
- notify only after successful publication to the relevant destination or destinations;
|
||||
- notification must be idempotent with respect to source id, digest, pipeline id, and destination id where applicable;
|
||||
- notification should not run for skipped or failed publications unless explicitly configured.
|
||||
|
||||
### `internal/logging`
|
||||
|
||||
Logging setup and helpers.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- centralize structured logging setup;
|
||||
- ensure logs omit secrets;
|
||||
- provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason.
|
||||
|
||||
## Deferred Ideas
|
||||
|
||||
The following are intentionally out of MVP unless separately accepted in a later roadmap:
|
||||
|
||||
- email, ntfy, Gotify, or other real notification adapters;
|
||||
- RSS/Atom feed generation;
|
||||
- PDF generation;
|
||||
- web UI;
|
||||
- full-text search;
|
||||
- dynamic plugin loading;
|
||||
- arbitrary transform chains;
|
||||
- workflow DAGs;
|
||||
- producer execution;
|
||||
- complex templating/theming;
|
||||
- bidirectional sync;
|
||||
- backup semantics.
|
||||
|
||||
## Key Invariants
|
||||
|
||||
- Producer apps own source bundle creation.
|
||||
- `distributor` owns destination publication state.
|
||||
- Source `manifest.json` is not copied as destination state.
|
||||
- Destination `.distributor.json` is the managed sentinel.
|
||||
- One pipeline has one source and one or more destinations.
|
||||
- Transform and publish policy are destination-specific.
|
||||
- Source files are canonical; HTML is derived.
|
||||
- Destructive replacement is allowed only inside managed destination bundle paths.
|
||||
- Core logic must be testable without real S3, SSH, or remote services.
|
||||
@@ -1,265 +0,0 @@
|
||||
# Storage Interface Roadmap
|
||||
|
||||
This roadmap defines the planned `internal/storage` contract for the `distributor` MVP. The goal is to give bundle validation, destination state inspection, publish planning, and backend adapters one consistent IO boundary without leaking local filesystem, SSH/SFTP, or S3-specific behavior into core packages.
|
||||
|
||||
## Purpose and Invariants
|
||||
|
||||
The storage layer is responsible for safe, backend-rooted access to files, objects, prefixes, and destination bundle paths.
|
||||
|
||||
Core invariants:
|
||||
|
||||
- Backends are opened at configured roots.
|
||||
- Core packages operate on backend-rooted logical paths, not absolute filesystem paths or raw object keys.
|
||||
- Backend adapters translate native storage behavior into common storage entries and typed errors.
|
||||
- Destructive operations remain narrow and managed.
|
||||
- Staging or atomic write behavior belongs behind the storage interface where practical.
|
||||
- The fake backend exists for tests only and must not be registered as a runtime backend.
|
||||
|
||||
## Logical Path Model
|
||||
|
||||
Storage paths are slash-separated logical paths relative to an already configured backend root.
|
||||
|
||||
File paths:
|
||||
|
||||
- must be non-empty;
|
||||
- must be relative;
|
||||
- must be clean;
|
||||
- must not contain `.` or `..` segments;
|
||||
- must not start with `/`;
|
||||
- must not contain backslashes;
|
||||
- must not resolve outside the backend root.
|
||||
|
||||
Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for traversal and destination emptiness checks.
|
||||
|
||||
Prefix matching must preserve logical path boundaries. A prefix of `foo` matches `foo` and entries below `foo/`; it must not match a sibling path such as `foobar`. Backends that map logical paths to object keys must apply the same normalized boundary rule after combining configured backend prefixes with caller-provided logical prefixes.
|
||||
|
||||
Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly.
|
||||
|
||||
## Core Interface Shape
|
||||
|
||||
The MVP should use a hybrid byte and stream interface:
|
||||
|
||||
```go
|
||||
type Backend interface {
|
||||
ReadFile(ctx context.Context, path string) ([]byte, error)
|
||||
OpenReader(ctx context.Context, path string) (io.ReadCloser, error)
|
||||
WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) (Entry, error)
|
||||
WriteFrom(ctx context.Context, path string, r io.Reader, opts WriteOptions) (Entry, error)
|
||||
Stat(ctx context.Context, path string) (Entry, error)
|
||||
Walk(ctx context.Context, prefix string, opts WalkOptions, fn WalkFunc) error
|
||||
HasAny(ctx context.Context, prefix string) (bool, error)
|
||||
DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts DeleteOptions) error
|
||||
}
|
||||
```
|
||||
|
||||
Byte helpers are expected to cover manifests, destination state, small source artifacts, and generated outputs. Stream methods are included from the start for backend flexibility and larger future artifacts.
|
||||
|
||||
Write operations should create required parent directories or prefixes as needed.
|
||||
|
||||
Concrete option and callback types should use this shape:
|
||||
|
||||
```go
|
||||
type WalkOptions struct {
|
||||
Recursive bool
|
||||
Limit int
|
||||
}
|
||||
|
||||
type WalkFunc func(Entry) error
|
||||
|
||||
var ErrStopWalk = errors.New("stop walk")
|
||||
|
||||
type WriteOptions struct {
|
||||
ContentType string
|
||||
Overwrite bool
|
||||
PreferAtomic bool
|
||||
Size int64
|
||||
SizeKnown bool
|
||||
}
|
||||
|
||||
type DeleteOptions struct {
|
||||
IgnoreMissing bool
|
||||
PruneEmptyDirs bool
|
||||
}
|
||||
```
|
||||
|
||||
`WalkOptions.Limit == 0` means no explicit limit. `SizeKnown` applies primarily to `WriteFrom`; byte writes can infer size from the provided data.
|
||||
|
||||
## Entries and Metadata
|
||||
|
||||
Storage metadata should be represented by an `Entry` model with at least:
|
||||
|
||||
- backend-relative logical path;
|
||||
- entry type;
|
||||
- size, where available.
|
||||
|
||||
Entry types:
|
||||
|
||||
- `file`: filesystem file or object-storage object;
|
||||
- `directory`: filesystem directory or logical prefix;
|
||||
- `symlink`: local filesystem symlink;
|
||||
- `other`: unknown or unsupported native entry type.
|
||||
|
||||
`Stat` returns metadata for one exact logical path. It may report a real filesystem directory, symlink, file, or exact object. It must not synthesize S3-like directory metadata solely because objects exist below a prefix; callers that need prefix existence or destination emptiness must use `HasAny` or `Walk`.
|
||||
|
||||
`Walk` traverses entries below a prefix and calls a callback for each entry. `WalkOptions` should include:
|
||||
|
||||
- whether traversal is recursive;
|
||||
- an optional entry limit for callers that only need to know whether content exists.
|
||||
|
||||
If a callback returns `ErrStopWalk`, traversal stops successfully and `Walk` returns nil. Any other callback error stops traversal and is returned with storage context where practical. If `WalkOptions.Limit` is greater than zero, reaching the limit stops traversal successfully.
|
||||
|
||||
Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal.
|
||||
|
||||
Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them.
|
||||
|
||||
`HasAny` reports whether at least one entry exists below a prefix. It should stop as soon as content is found.
|
||||
|
||||
Source validation must reject symlink entries reported by local `Stat` or `Walk`.
|
||||
|
||||
## Read Behavior
|
||||
|
||||
`ReadFile` reads the whole object into memory and is appropriate for MVP manifest, state, and ordinary artifact handling.
|
||||
|
||||
`OpenReader` returns a stream for callers that need to copy or hash content without requiring a second storage-specific API. Callers must close the returned reader.
|
||||
|
||||
Both read methods must:
|
||||
|
||||
- validate logical paths before backend access;
|
||||
- reject directories, prefixes, symlinks, and unsupported entries;
|
||||
- return typed not-found and invalid-path errors where applicable.
|
||||
|
||||
## Write Behavior
|
||||
|
||||
`WriteOptions` should include:
|
||||
|
||||
- content type, when the destination backend can use it;
|
||||
- overwrite permission;
|
||||
- atomic or staged write preference;
|
||||
- optional known size for stream writes.
|
||||
|
||||
Backends own staging and atomic behavior where practical:
|
||||
|
||||
- Local backend writes to a temporary file in the destination directory and renames or promotes into place.
|
||||
- SSH/SFTP backend should use a temporary remote file and rename where available.
|
||||
- S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata.
|
||||
|
||||
Remote adapters may buffer or spool `WriteFrom` input when needed to satisfy backend requirements such as content length, multipart upload, or retry behavior. Callers that know the stream size should set `SizeKnown` and `Size`.
|
||||
|
||||
If overwrite is false and the target exists, writes should fail with an already-exists error.
|
||||
|
||||
`WriteFile` and `WriteFrom` should return the written `Entry`, including final path and size where available.
|
||||
|
||||
`DeleteOptions` should include:
|
||||
|
||||
- whether missing managed output paths are ignored;
|
||||
- whether empty parent directories may be pruned for filesystem-like backends.
|
||||
|
||||
## Managed Deletion
|
||||
|
||||
The storage interface should expose a guarded managed deletion operation rather than raw recursive delete.
|
||||
|
||||
`DeleteManagedBundle(ctx, bundlePath, managedOutputPaths, opts)` may delete only:
|
||||
|
||||
- files or objects listed in valid `.distributor.json.outputs`;
|
||||
- `.distributor.json` at the destination bundle path;
|
||||
- empty directories created by those files, for filesystem-like backends.
|
||||
|
||||
`managedOutputPaths` are relative to the destination bundle path. The backend validates each path and resolves it under `bundlePath`.
|
||||
|
||||
If `bundlePath == ""`, deletion may remove explicit managed files at the destination root, but must never delete the root itself.
|
||||
|
||||
Prefix or recursive deletion is out of MVP scope. A future force-overwrite stage may add broader behavior, but it must remain explicit and separately documented.
|
||||
|
||||
## Destination Emptiness
|
||||
|
||||
Destination emptiness should use `HasAny(prefix)` and typed not-found behavior. Callers that only need emptiness must not materialize a full recursive traversal.
|
||||
|
||||
Rules:
|
||||
|
||||
- A local destination bundle path is empty when the directory does not exist or exists with no entries.
|
||||
- An S3-compatible prefix is empty when no objects exist below that exact destination bundle prefix.
|
||||
- Entries outside the exact destination bundle path or prefix do not affect emptiness.
|
||||
|
||||
## Error Model
|
||||
|
||||
Storage should expose typed error categories with wrapping context. Callers should use helper predicates rather than string matching.
|
||||
|
||||
Required categories:
|
||||
|
||||
- not found;
|
||||
- already exists;
|
||||
- not empty;
|
||||
- invalid path;
|
||||
- conflict;
|
||||
- permission;
|
||||
- temporary;
|
||||
- unsupported;
|
||||
- unknown.
|
||||
|
||||
Adapters should translate backend-native errors into these categories while preserving useful operation, backend, path, and cause context.
|
||||
|
||||
## Adapter Expectations
|
||||
|
||||
### Local
|
||||
|
||||
The local backend should:
|
||||
|
||||
- constrain all operations beneath the configured root;
|
||||
- reject traversal and absolute logical paths;
|
||||
- report symlinks through metadata;
|
||||
- reject symlink reads for source artifacts;
|
||||
- use staged writes where practical;
|
||||
- perform managed deletion only for explicit managed files and `.distributor.json`;
|
||||
- clean up empty directories created by managed outputs where safe.
|
||||
|
||||
### Fake
|
||||
|
||||
The fake backend should:
|
||||
|
||||
- be in-memory and deterministic;
|
||||
- implement the same logical path validation rules;
|
||||
- support `Stat`, `Walk`, `HasAny`, byte reads and writes, stream reads and writes, managed deletion, and destination emptiness helper behavior;
|
||||
- support configured symlink entries for validation tests;
|
||||
- be used only by tests.
|
||||
|
||||
### SSH/SFTP
|
||||
|
||||
The SSH/SFTP backend should:
|
||||
|
||||
- use native SFTP operations;
|
||||
- enforce the same logical path rules as local storage;
|
||||
- use temporary file plus rename for staged writes where available;
|
||||
- translate remote errors into storage error categories;
|
||||
- avoid exposing SSH or SFTP dependency types through `internal/storage`.
|
||||
|
||||
### S3-Compatible
|
||||
|
||||
The S3-compatible backend should:
|
||||
|
||||
- treat prefixes as object trees, not real directories;
|
||||
- normalize configured prefix plus logical path into object keys;
|
||||
- use object PUT as publish-on-success;
|
||||
- set content type from `WriteOptions`;
|
||||
- implement traversal and emptiness by exact prefix;
|
||||
- use backend pagination for traversal where available;
|
||||
- allow `HasAny` to stop after the first matching object;
|
||||
- constrain managed deletion to listed output objects and `.distributor.json`.
|
||||
|
||||
## Tests and Fixtures
|
||||
|
||||
Storage implementation stages should test:
|
||||
|
||||
- path validation rejects absolute paths, traversal, empty file paths, backslashes, and dot segments;
|
||||
- `Walk` visits backend-rooted logical paths under a prefix and supports recursive traversal;
|
||||
- a materializing helper sorts walk results lexically for deterministic tests and CLI output;
|
||||
- `HasAny` returns quickly for non-empty prefixes without requiring full traversal;
|
||||
- `ReadFile` and `OpenReader` return equivalent bytes;
|
||||
- `WriteFile` and `WriteFrom` honor overwrite and content-type options;
|
||||
- local staged writes do not leave final files on failure where testable;
|
||||
- managed deletion deletes only state-listed files and `.distributor.json`;
|
||||
- managed deletion never deletes destination root or unlisted files;
|
||||
- destination emptiness helper handles missing, empty, and non-empty local paths;
|
||||
- S3-compatible traversal can use pagination without loading a whole prefix into memory;
|
||||
- symlink entries are reported and rejected by source validation;
|
||||
- typed errors are usable through helper predicates;
|
||||
- fake backend behavior matches local backend semantics relevant to core tests.
|
||||
Reference in New Issue
Block a user