Update roadmap documentation to remove completed work
This commit is contained in:
@@ -1,580 +0,0 @@
|
|||||||
# Distributor Configuration Roadmap
|
|
||||||
|
|
||||||
This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.
|
|
||||||
|
|
||||||
## Configuration Goals
|
|
||||||
|
|
||||||
The MVP configuration should be:
|
|
||||||
|
|
||||||
- explicit enough to avoid hidden publication behavior;
|
|
||||||
- compact enough for routine self-hosted use;
|
|
||||||
- backend-agnostic at the pipeline layer;
|
|
||||||
- capable of local, SSH/SFTP, and S3-compatible source and destination backends;
|
|
||||||
- ready for future notification adapters without exposing a fake notification feature in the MVP.
|
|
||||||
|
|
||||||
## Top-Level Shape
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
pipelines:
|
|
||||||
- id: weather-daily
|
|
||||||
source:
|
|
||||||
backend: local
|
|
||||||
path: /var/spool/distributor/weather
|
|
||||||
|
|
||||||
validation:
|
|
||||||
on_digest_mismatch: fail
|
|
||||||
|
|
||||||
destinations:
|
|
||||||
- id: markdown-archive
|
|
||||||
backend: s3
|
|
||||||
endpoint: https://s3.example.com
|
|
||||||
bucket: reports
|
|
||||||
prefix: weather/archive
|
|
||||||
region: us-east-1
|
|
||||||
force_path_style: true
|
|
||||||
credentials:
|
|
||||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
|
||||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
|
||||||
|
|
||||||
publish:
|
|
||||||
source: true
|
|
||||||
html: false
|
|
||||||
|
|
||||||
transfer:
|
|
||||||
on_destination_same: skip
|
|
||||||
on_destination_older: replace
|
|
||||||
on_destination_newer: skip
|
|
||||||
on_conflict: fail
|
|
||||||
|
|
||||||
- id: static-site
|
|
||||||
backend: ssh
|
|
||||||
uri: ssh://deploy@example.com:22
|
|
||||||
path: /srv/www/weather
|
|
||||||
|
|
||||||
publish:
|
|
||||||
source: false
|
|
||||||
html: true
|
|
||||||
|
|
||||||
transform:
|
|
||||||
markdown_to_html:
|
|
||||||
enabled: true
|
|
||||||
mode: sidecar
|
|
||||||
|
|
||||||
transfer:
|
|
||||||
on_destination_same: skip
|
|
||||||
on_destination_older: replace
|
|
||||||
on_destination_newer: skip
|
|
||||||
on_conflict: fail
|
|
||||||
```
|
|
||||||
|
|
||||||
## Pipeline Fields
|
|
||||||
|
|
||||||
Each pipeline must include:
|
|
||||||
|
|
||||||
- `id`: Required stable pipeline identifier.
|
|
||||||
- `source`: Required source backend configuration.
|
|
||||||
- `destinations`: Required non-empty list of destination configurations.
|
|
||||||
|
|
||||||
Optional pipeline-level fields:
|
|
||||||
|
|
||||||
- `validation`: Source validation behavior.
|
|
||||||
- Future: `notifications` or `notify`, when notification adapters are implemented.
|
|
||||||
|
|
||||||
A pipeline has exactly one source and one or more destinations.
|
|
||||||
|
|
||||||
## Pipeline ID Rules
|
|
||||||
|
|
||||||
`pipelines[].id` should:
|
|
||||||
|
|
||||||
- be required;
|
|
||||||
- be unique across the config file;
|
|
||||||
- be stable over time;
|
|
||||||
- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`.
|
|
||||||
|
|
||||||
Recommended validation:
|
|
||||||
|
|
||||||
```text
|
|
||||||
^[a-zA-Z0-9][a-zA-Z0-9._-]*$
|
|
||||||
```
|
|
||||||
|
|
||||||
## Source Configuration
|
|
||||||
|
|
||||||
`source` defines the source root where bundles are discovered.
|
|
||||||
|
|
||||||
The source backend may be:
|
|
||||||
|
|
||||||
- `local`;
|
|
||||||
- `ssh`;
|
|
||||||
- `s3`.
|
|
||||||
|
|
||||||
The source is scanned for `manifest.json` files beneath the configured root.
|
|
||||||
|
|
||||||
### Local source
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
source:
|
|
||||||
backend: local
|
|
||||||
path: /var/spool/distributor/weather
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields:
|
|
||||||
|
|
||||||
- `backend: local`
|
|
||||||
- `path`
|
|
||||||
|
|
||||||
### SSH source
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
source:
|
|
||||||
backend: ssh
|
|
||||||
uri: ssh://reports@example.com:22
|
|
||||||
path: /var/spool/distributor/weather
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields:
|
|
||||||
|
|
||||||
- `backend: ssh`
|
|
||||||
- `uri`
|
|
||||||
- `path`
|
|
||||||
|
|
||||||
Recommended authentication behavior:
|
|
||||||
|
|
||||||
- use SSH agent by default;
|
|
||||||
- use local known_hosts validation by default;
|
|
||||||
- support optional key file configuration later if needed;
|
|
||||||
- do not require passwords in YAML.
|
|
||||||
|
|
||||||
Optional future fields:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
known_hosts: /home/user/.ssh/known_hosts
|
|
||||||
key_file: /home/user/.ssh/id_ed25519
|
|
||||||
```
|
|
||||||
|
|
||||||
### S3 source
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
source:
|
|
||||||
backend: s3
|
|
||||||
endpoint: https://s3.example.com
|
|
||||||
bucket: reports
|
|
||||||
prefix: incoming/weather
|
|
||||||
region: us-east-1
|
|
||||||
force_path_style: true
|
|
||||||
credentials:
|
|
||||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
|
||||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields:
|
|
||||||
|
|
||||||
- `backend: s3`
|
|
||||||
- `endpoint`
|
|
||||||
- `bucket`
|
|
||||||
|
|
||||||
Optional fields:
|
|
||||||
|
|
||||||
- `prefix`
|
|
||||||
- `region`
|
|
||||||
- `force_path_style`
|
|
||||||
- `credentials`
|
|
||||||
|
|
||||||
Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.
|
|
||||||
|
|
||||||
## Destination Configuration
|
|
||||||
|
|
||||||
Each destination defines one publication target for a pipeline.
|
|
||||||
|
|
||||||
Required destination fields:
|
|
||||||
|
|
||||||
- `id`
|
|
||||||
- `backend`
|
|
||||||
- backend-specific location fields;
|
|
||||||
- `publish`
|
|
||||||
|
|
||||||
Optional destination fields:
|
|
||||||
|
|
||||||
- `transform`
|
|
||||||
- `transfer`
|
|
||||||
|
|
||||||
Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.
|
|
||||||
|
|
||||||
## Destination ID Rules
|
|
||||||
|
|
||||||
`destinations[].id` should:
|
|
||||||
|
|
||||||
- be required;
|
|
||||||
- be unique within the containing pipeline;
|
|
||||||
- be stable over time;
|
|
||||||
- use a slug-like format.
|
|
||||||
|
|
||||||
Recommended examples:
|
|
||||||
|
|
||||||
- `markdown-archive`
|
|
||||||
- `static-site`
|
|
||||||
- `full-mirror`
|
|
||||||
|
|
||||||
## Local Destination
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
destinations:
|
|
||||||
- id: local-static
|
|
||||||
backend: local
|
|
||||||
path: /srv/www/reports
|
|
||||||
publish:
|
|
||||||
source: false
|
|
||||||
html: true
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields:
|
|
||||||
|
|
||||||
- `backend: local`
|
|
||||||
- `path`
|
|
||||||
- `publish`
|
|
||||||
|
|
||||||
## SSH Destination
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
destinations:
|
|
||||||
- id: static-site
|
|
||||||
backend: ssh
|
|
||||||
uri: ssh://deploy@example.com:22
|
|
||||||
path: /srv/www/weather
|
|
||||||
publish:
|
|
||||||
source: false
|
|
||||||
html: true
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields:
|
|
||||||
|
|
||||||
- `backend: ssh`
|
|
||||||
- `uri`
|
|
||||||
- `path`
|
|
||||||
- `publish`
|
|
||||||
|
|
||||||
The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`.
|
|
||||||
|
|
||||||
## S3 Destination
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
destinations:
|
|
||||||
- id: markdown-archive
|
|
||||||
backend: s3
|
|
||||||
endpoint: https://s3.example.com
|
|
||||||
bucket: reports
|
|
||||||
prefix: weather/archive
|
|
||||||
region: us-east-1
|
|
||||||
force_path_style: true
|
|
||||||
credentials:
|
|
||||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
|
||||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
|
||||||
publish:
|
|
||||||
source: true
|
|
||||||
html: false
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields:
|
|
||||||
|
|
||||||
- `backend: s3`
|
|
||||||
- `endpoint`
|
|
||||||
- `bucket`
|
|
||||||
- `publish`
|
|
||||||
|
|
||||||
Optional fields:
|
|
||||||
|
|
||||||
- `prefix`
|
|
||||||
- `region`
|
|
||||||
- `force_path_style`
|
|
||||||
- `credentials`
|
|
||||||
|
|
||||||
## Backend Configuration Normalization
|
|
||||||
|
|
||||||
The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.
|
|
||||||
|
|
||||||
Validation should catch:
|
|
||||||
|
|
||||||
- missing backend names;
|
|
||||||
- unsupported backend names;
|
|
||||||
- missing backend-specific required fields;
|
|
||||||
- duplicate pipeline IDs;
|
|
||||||
- duplicate destination IDs within a pipeline;
|
|
||||||
- empty destination lists;
|
|
||||||
- invalid policy values.
|
|
||||||
|
|
||||||
## Publication Policy
|
|
||||||
|
|
||||||
`publish` controls which categories of files are written to a destination.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
publish:
|
|
||||||
source: true
|
|
||||||
html: false
|
|
||||||
```
|
|
||||||
|
|
||||||
Fields:
|
|
||||||
|
|
||||||
- `source`: Publish source artifacts listed in `manifest.json`.
|
|
||||||
- `html`: Publish HTML files generated from Markdown source artifacts.
|
|
||||||
|
|
||||||
At least one of `source` or `html` must be true.
|
|
||||||
|
|
||||||
Recommended defaults:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
publish:
|
|
||||||
source: true
|
|
||||||
html: false
|
|
||||||
```
|
|
||||||
|
|
||||||
No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP.
|
|
||||||
|
|
||||||
## Transform Configuration
|
|
||||||
|
|
||||||
For MVP, the only supported transform is Markdown to HTML.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
transform:
|
|
||||||
markdown_to_html:
|
|
||||||
enabled: true
|
|
||||||
mode: sidecar
|
|
||||||
```
|
|
||||||
|
|
||||||
Fields:
|
|
||||||
|
|
||||||
- `enabled`: Whether Markdown-to-HTML transform is enabled.
|
|
||||||
- `mode`: Output mode. MVP value: `sidecar`.
|
|
||||||
|
|
||||||
MVP `sidecar` behavior:
|
|
||||||
|
|
||||||
- each listed Markdown source file generates an HTML file with the same base path and `.html` extension;
|
|
||||||
- `report.md` generates `report.html`;
|
|
||||||
- generated files are destination publication artifacts;
|
|
||||||
- source bundles are not mutated.
|
|
||||||
|
|
||||||
MVP defaulting:
|
|
||||||
|
|
||||||
- If `publish.html` is false, transform may be omitted.
|
|
||||||
- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail.
|
|
||||||
|
|
||||||
## Validation Policy
|
|
||||||
|
|
||||||
Pipeline-level validation is intentionally narrow in the MVP.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
validation:
|
|
||||||
on_digest_mismatch: fail
|
|
||||||
```
|
|
||||||
|
|
||||||
Supported value:
|
|
||||||
|
|
||||||
- `fail`
|
|
||||||
|
|
||||||
Default:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
on_digest_mismatch: fail
|
|
||||||
```
|
|
||||||
|
|
||||||
Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.
|
|
||||||
|
|
||||||
## Transfer Policy
|
|
||||||
|
|
||||||
Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
transfer:
|
|
||||||
on_destination_same: skip
|
|
||||||
on_destination_older: replace
|
|
||||||
on_destination_newer: skip
|
|
||||||
on_conflict: fail
|
|
||||||
```
|
|
||||||
|
|
||||||
Supported fields:
|
|
||||||
|
|
||||||
- `on_destination_same`
|
|
||||||
- `on_destination_older`
|
|
||||||
- `on_destination_newer`
|
|
||||||
- `on_conflict`
|
|
||||||
|
|
||||||
MVP supported values are intentionally limited by field:
|
|
||||||
|
|
||||||
- `on_destination_same`: `skip` or `fail`
|
|
||||||
- `on_destination_older`: `replace` or `fail`
|
|
||||||
- `on_destination_newer`: `skip` or `fail`
|
|
||||||
- `on_conflict`: `fail`
|
|
||||||
|
|
||||||
Recommended MVP defaults:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
transfer:
|
|
||||||
on_destination_same: skip
|
|
||||||
on_destination_older: replace
|
|
||||||
on_destination_newer: skip
|
|
||||||
on_conflict: fail
|
|
||||||
```
|
|
||||||
|
|
||||||
Safety rule:
|
|
||||||
|
|
||||||
- `replace` must never perform broad deletion against a destination root.
|
|
||||||
- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
|
|
||||||
- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
|
|
||||||
- Broader replacement values, including replacing newer destinations or conflicts, are deferred to a later explicit force-overwrite stage.
|
|
||||||
|
|
||||||
## Path Mapping
|
|
||||||
|
|
||||||
MVP path mapping is fixed:
|
|
||||||
|
|
||||||
```text
|
|
||||||
destination bundle path = destination root + source relative bundle path
|
|
||||||
```
|
|
||||||
|
|
||||||
Example:
|
|
||||||
|
|
||||||
```text
|
|
||||||
source root: /var/spool/reports
|
|
||||||
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
|
|
||||||
relative bundle path: weather/daily/brentwood/2026-05-30
|
|
||||||
|
|
||||||
destination root: /srv/www/reports
|
|
||||||
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30
|
|
||||||
```
|
|
||||||
|
|
||||||
Future config may support explicit path mapping, but MVP should not.
|
|
||||||
|
|
||||||
## Dry Run Configuration and CLI Behavior
|
|
||||||
|
|
||||||
Dry-run should be a CLI flag rather than a persistent config setting.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
distributor run --config config.yml --dry-run
|
|
||||||
```
|
|
||||||
|
|
||||||
Dry-run should report:
|
|
||||||
|
|
||||||
- pipeline ID;
|
|
||||||
- source backend;
|
|
||||||
- destination ID;
|
|
||||||
- destination backend;
|
|
||||||
- discovered bundle ID;
|
|
||||||
- relative bundle path;
|
|
||||||
- planned action;
|
|
||||||
- reason;
|
|
||||||
- transform outputs that would be generated;
|
|
||||||
- files that would be written or deleted.
|
|
||||||
|
|
||||||
## Example: Weather Pipeline
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
pipelines:
|
|
||||||
- id: weather-daily
|
|
||||||
source:
|
|
||||||
backend: local
|
|
||||||
path: /var/spool/distributor/weather
|
|
||||||
|
|
||||||
validation:
|
|
||||||
on_digest_mismatch: fail
|
|
||||||
|
|
||||||
destinations:
|
|
||||||
- id: markdown-archive
|
|
||||||
backend: s3
|
|
||||||
endpoint: https://s3.example.com
|
|
||||||
bucket: reports
|
|
||||||
prefix: weather/archive
|
|
||||||
region: us-east-1
|
|
||||||
force_path_style: true
|
|
||||||
credentials:
|
|
||||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
|
||||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
|
||||||
publish:
|
|
||||||
source: true
|
|
||||||
html: false
|
|
||||||
|
|
||||||
- id: static-site
|
|
||||||
backend: ssh
|
|
||||||
uri: ssh://deploy@web.example.com:22
|
|
||||||
path: /srv/www/weather
|
|
||||||
publish:
|
|
||||||
source: false
|
|
||||||
html: true
|
|
||||||
transform:
|
|
||||||
markdown_to_html:
|
|
||||||
enabled: true
|
|
||||||
mode: sidecar
|
|
||||||
```
|
|
||||||
|
|
||||||
## Example: D&D Recap Pipeline
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
pipelines:
|
|
||||||
- id: dnd-session-recaps
|
|
||||||
source:
|
|
||||||
backend: local
|
|
||||||
path: /var/spool/distributor/dnd/session-recaps
|
|
||||||
|
|
||||||
destinations:
|
|
||||||
- id: private-markdown-archive
|
|
||||||
backend: s3
|
|
||||||
endpoint: https://s3.example.com
|
|
||||||
bucket: reports
|
|
||||||
prefix: dnd/session-recaps
|
|
||||||
region: us-east-1
|
|
||||||
force_path_style: true
|
|
||||||
credentials:
|
|
||||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
|
||||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
|
||||||
publish:
|
|
||||||
source: true
|
|
||||||
html: false
|
|
||||||
|
|
||||||
- id: private-html-site
|
|
||||||
backend: local
|
|
||||||
path: /srv/www/private/dnd/session-recaps
|
|
||||||
publish:
|
|
||||||
source: false
|
|
||||||
html: true
|
|
||||||
transform:
|
|
||||||
markdown_to_html:
|
|
||||||
enabled: true
|
|
||||||
mode: sidecar
|
|
||||||
```
|
|
||||||
|
|
||||||
## Future Notification Configuration
|
|
||||||
|
|
||||||
Notification should not be exposed as a functional MVP feature unless an adapter exists.
|
|
||||||
|
|
||||||
The internal pipeline may include a no-op notification stage. Future config may look like:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
notifications:
|
|
||||||
- id: weather-email
|
|
||||||
backend: email
|
|
||||||
after_destinations:
|
|
||||||
- static-site
|
|
||||||
subject: "Weather report published"
|
|
||||||
```
|
|
||||||
|
|
||||||
Future notification policy should require:
|
|
||||||
|
|
||||||
- notification after successful relevant publication;
|
|
||||||
- idempotency by source manifest id and digest;
|
|
||||||
- no duplicate notification unless explicitly forced.
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
|
|
||||||
2. Implement config loading and strict validation.
|
|
||||||
3. Implement backend-specific config validation for local, SSH, and S3.
|
|
||||||
4. Implement defaulting for validation and transfer policies.
|
|
||||||
5. Require explicit transform configuration when `publish.html` is true.
|
|
||||||
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
|
|
||||||
7. Connect config to backend registry and publish planner.
|
|
||||||
8. Add `--pipeline` filtering for targeted runs.
|
|
||||||
9. Add `--dry-run` output that reflects the resolved config and planned actions.
|
|
||||||
|
|
||||||
## Deferred Configuration
|
|
||||||
|
|
||||||
The following configuration ideas are intentionally outside the MVP:
|
|
||||||
|
|
||||||
- warning-only digest mismatch handling;
|
|
||||||
- unmanaged destination overwrite flags until the explicit force-overwrite roadmap stage;
|
|
||||||
- force replacement of destinations with different source ids until the explicit force-overwrite roadmap stage.
|
|
||||||
@@ -1,398 +0,0 @@
|
|||||||
# Distributor Contracts Roadmap
|
|
||||||
|
|
||||||
This roadmap defines the contracts that `distributor` should implement before or alongside the MVP. The goal is to make bundle validation, destination state, digest verification, and safe replacement deterministic and testable before backend-specific publication behavior is layered on top.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
`distributor` publishes manifested report bundles produced by other applications. Producer applications own domain-specific report generation. `distributor` owns validation, optional transformation, destination publication, and destination state.
|
|
||||||
|
|
||||||
The MVP contract has two durable files:
|
|
||||||
|
|
||||||
- `manifest.json`: source-owned bundle manifest produced by the upstream application.
|
|
||||||
- `.distributor.json`: destination-owned publication state written by `distributor`.
|
|
||||||
|
|
||||||
`manifest.json` is not copied to the destination as destination state. Instead, `.distributor.json` records the normalized source manifest, generated output metadata, and distributor-owned publication metadata.
|
|
||||||
|
|
||||||
All manifest and state timestamps should be serialized as RFC3339. Internal comparison should use parsed timestamp values, and distributor-written timestamps should be normalized to RFC3339 UTC.
|
|
||||||
|
|
||||||
## Terminology
|
|
||||||
|
|
||||||
- **Source root**: Configured root path for a pipeline source.
|
|
||||||
- **Bundle root**: Directory beneath the source root that contains `manifest.json`.
|
|
||||||
- **Relative bundle path**: Bundle root path relative to the source root.
|
|
||||||
- **Destination root**: Configured root path or prefix for a destination.
|
|
||||||
- **Destination bundle path**: Destination root plus the relative bundle path, unless a future mapping option overrides that behavior.
|
|
||||||
- **Source artifact**: File listed in the source `manifest.json`.
|
|
||||||
- **Generated artifact**: File created by `distributor`, such as an HTML file derived from Markdown.
|
|
||||||
- **Destination state**: `.distributor.json` at the destination bundle path.
|
|
||||||
|
|
||||||
## Source Bundle Contract
|
|
||||||
|
|
||||||
A source bundle is a directory containing a `manifest.json` file. For MVP, the manifest schema is intentionally minimal.
|
|
||||||
|
|
||||||
### Required `manifest.json` fields
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema_version": 1,
|
|
||||||
"id": "weather.daily.brentwood.2026-05-30",
|
|
||||||
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
|
||||||
"created": "2026-05-30T11:10:00Z",
|
|
||||||
"files": [
|
|
||||||
{
|
|
||||||
"path": "report.md",
|
|
||||||
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
|
||||||
"size": 12345
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Required top-level fields:
|
|
||||||
|
|
||||||
- `schema_version`: Source manifest schema version. MVP value: `1`.
|
|
||||||
- `id`: Stable bundle identifier. Required, non-empty string.
|
|
||||||
- `digest`: Bundle digest. Required, lowercase `sha256:<64 hex>` string.
|
|
||||||
- `created`: Bundle creation timestamp. Required RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
|
|
||||||
- `files`: Non-empty array of file objects.
|
|
||||||
|
|
||||||
Required file fields:
|
|
||||||
|
|
||||||
- `path`: Relative path from bundle root to source artifact.
|
|
||||||
- `sha256`: Per-file digest as lowercase `sha256:<64 hex>`.
|
|
||||||
- `size`: File size in bytes.
|
|
||||||
|
|
||||||
No other source manifest fields are required for the MVP. Additional fields may be ignored unless later documented. Destination state records the normalized source manifest model, not raw unknown manifest fields.
|
|
||||||
|
|
||||||
## Source Path Safety Rules
|
|
||||||
|
|
||||||
For every `files[].path`:
|
|
||||||
|
|
||||||
- Path must be relative.
|
|
||||||
- Path must not be empty.
|
|
||||||
- Path must not contain `..` segments.
|
|
||||||
- Path must not resolve outside the bundle root.
|
|
||||||
- Path must use slash-separated logical paths in the manifest.
|
|
||||||
- Absolute paths are invalid.
|
|
||||||
- Symlinks should be rejected for MVP unless a later policy deliberately supports them.
|
|
||||||
- `manifest.json` itself should not be listed as a source artifact.
|
|
||||||
- `.distributor.json` should not be listed as a source artifact.
|
|
||||||
- Duplicate logical file paths are invalid after path normalization.
|
|
||||||
|
|
||||||
The implementation should validate paths before reading file contents.
|
|
||||||
|
|
||||||
## Digest Contract
|
|
||||||
|
|
||||||
The MVP validates both per-file digests and the bundle digest.
|
|
||||||
|
|
||||||
### Per-file digest
|
|
||||||
|
|
||||||
For each file listed in `files`, compute:
|
|
||||||
|
|
||||||
```text
|
|
||||||
sha256(file bytes)
|
|
||||||
```
|
|
||||||
|
|
||||||
The computed digest must match `files[].sha256`.
|
|
||||||
|
|
||||||
The actual file size must match `files[].size`.
|
|
||||||
|
|
||||||
### Bundle digest
|
|
||||||
|
|
||||||
The bundle digest is computed from the listed file records in the listed order. The canonical algorithm is:
|
|
||||||
|
|
||||||
1. For each file listed in `files`, in order:
|
|
||||||
- validate the file path;
|
|
||||||
- compute the file SHA256;
|
|
||||||
- determine the file size.
|
|
||||||
2. Construct a canonical JSON array containing only:
|
|
||||||
- `path`;
|
|
||||||
- `sha256`;
|
|
||||||
- `size`.
|
|
||||||
3. Preserve the source manifest's file order.
|
|
||||||
4. Encode the array deterministically with:
|
|
||||||
- object fields in exactly this order: `path`, `sha256`, `size`;
|
|
||||||
- no extra spaces;
|
|
||||||
- no trailing newline;
|
|
||||||
- lowercase `sha256:<64 hex>` digest strings.
|
|
||||||
5. Compute `sha256(canonical JSON bytes)`.
|
|
||||||
6. Compare the result to top-level `digest`.
|
|
||||||
|
|
||||||
Conceptual canonical payload:
|
|
||||||
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{"path":"report.md","sha256":"sha256:...","size":12345},
|
|
||||||
{"path":"summary.txt","sha256":"sha256:...","size":234}
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
This avoids ambiguous concatenation of file bytes while keeping the manifest small.
|
|
||||||
|
|
||||||
Implementation fixtures should include at least one reference manifest and canonical payload with known per-file and bundle digests.
|
|
||||||
|
|
||||||
## Digest Mismatch Behavior
|
|
||||||
|
|
||||||
Digest validation is always fatal in the MVP. A digest mismatch must fail the affected bundle, pipeline, and run before any destination writes occur.
|
|
||||||
|
|
||||||
Warning-only digest behavior is intentionally deferred until a later roadmap accepts transitional ingestion semantics.
|
|
||||||
|
|
||||||
## Bundle Discovery Contract
|
|
||||||
|
|
||||||
A source path may contain one bundle or a tree of bundles. Discovery should scan beneath the configured source root for `manifest.json` files.
|
|
||||||
|
|
||||||
For each discovered manifest:
|
|
||||||
|
|
||||||
- Bundle root is the directory containing `manifest.json`.
|
|
||||||
- Relative bundle path is computed relative to source root.
|
|
||||||
- Destination bundle path is destination root plus relative bundle path, unless a future mapping option overrides it.
|
|
||||||
|
|
||||||
If nested manifests are found, the MVP should fail with a clear error unless a later policy defines nested-bundle semantics.
|
|
||||||
|
|
||||||
## Destination State Contract
|
|
||||||
|
|
||||||
Destinations are managed by `.distributor.json`, not by copying `manifest.json`.
|
|
||||||
|
|
||||||
A destination bundle path is considered distributor-managed only when it contains a valid `.distributor.json` written by `distributor`. Force or unmanaged-overwrite behavior is not part of the MVP.
|
|
||||||
|
|
||||||
### Required `.distributor.json` shape
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema_version": 1,
|
|
||||||
"distributor_version": "0.1.0",
|
|
||||||
"pipeline_id": "weather-daily",
|
|
||||||
"destination_id": "static-site",
|
|
||||||
"published_at": "2026-05-30T11:12:00Z",
|
|
||||||
"source": {
|
|
||||||
"manifest": {
|
|
||||||
"schema_version": 1,
|
|
||||||
"id": "weather.daily.brentwood.2026-05-30",
|
|
||||||
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
|
||||||
"created": "2026-05-30T11:10:00Z",
|
|
||||||
"files": [
|
|
||||||
{
|
|
||||||
"path": "report.md",
|
|
||||||
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
|
||||||
"size": 12345
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"path": "report.html",
|
|
||||||
"kind": "generated",
|
|
||||||
"source_path": "report.md",
|
|
||||||
"transform": "markdown_to_html",
|
|
||||||
"sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333",
|
|
||||||
"size": 23456
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields:
|
|
||||||
|
|
||||||
- `schema_version`: Destination state schema version. MVP value: `1`.
|
|
||||||
- `distributor_version`: Optional diagnostic distributor version. It must not affect source comparison.
|
|
||||||
- `pipeline_id`: Pipeline that produced the destination publication.
|
|
||||||
- `destination_id`: Destination within the pipeline.
|
|
||||||
- `published_at`: RFC3339 timestamp.
|
|
||||||
- `source.manifest`: Normalized source manifest model used for this publication.
|
|
||||||
- `outputs`: Array of files written by `distributor` for this destination.
|
|
||||||
|
|
||||||
Required output fields:
|
|
||||||
|
|
||||||
- `path`: Destination-relative output path within the destination bundle path.
|
|
||||||
- `kind`: `source` or `generated`.
|
|
||||||
- `source_path`: Source artifact path that produced this output. For copied source files, this should equal `path` unless renamed by a future feature.
|
|
||||||
- `transform`: Transform identifier for generated files. Empty or omitted may be allowed for copied source files.
|
|
||||||
- `sha256`: Output file digest as lowercase `sha256:<64 hex>`.
|
|
||||||
- `size`: Output file size in bytes.
|
|
||||||
|
|
||||||
## Destination Comparison Rules
|
|
||||||
|
|
||||||
Destination comparison uses `.distributor.json`, not destination `manifest.json`.
|
|
||||||
|
|
||||||
For a source bundle and destination bundle path:
|
|
||||||
|
|
||||||
### No `.distributor.json`
|
|
||||||
|
|
||||||
If no `.distributor.json` exists and the destination path is empty:
|
|
||||||
|
|
||||||
- Publish normally.
|
|
||||||
|
|
||||||
If no `.distributor.json` exists and the destination path is non-empty:
|
|
||||||
|
|
||||||
- Fail as unmanaged content.
|
|
||||||
|
|
||||||
Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix.
|
|
||||||
|
|
||||||
### Pipeline or destination mismatch
|
|
||||||
|
|
||||||
If `.distributor.json` exists but its `pipeline_id` or `destination_id` differs from the current pipeline or destination config:
|
|
||||||
|
|
||||||
- Fail as a conflict.
|
|
||||||
|
|
||||||
### Same source manifest
|
|
||||||
|
|
||||||
If `.distributor.json` exists and `source.manifest` exactly matches the current normalized source manifest:
|
|
||||||
|
|
||||||
- Skip as already published.
|
|
||||||
|
|
||||||
### Same source id, destination older
|
|
||||||
|
|
||||||
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is older than the source `created`:
|
|
||||||
|
|
||||||
- Replace destination contents, subject to replacement safety rules.
|
|
||||||
|
|
||||||
### Same source id, destination newer
|
|
||||||
|
|
||||||
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is newer than the source `created`:
|
|
||||||
|
|
||||||
- Skip and log that destination is newer than source.
|
|
||||||
|
|
||||||
### Same source id and created, different digest
|
|
||||||
|
|
||||||
If `.distributor.json` exists, `source.manifest.id` and `created` match but `digest` differs:
|
|
||||||
|
|
||||||
- Fail as a conflict.
|
|
||||||
|
|
||||||
### Different source id
|
|
||||||
|
|
||||||
If `.distributor.json` exists and `source.manifest.id` differs from the source manifest `id`:
|
|
||||||
|
|
||||||
- Fail as a conflict.
|
|
||||||
|
|
||||||
## Replacement Safety Rules
|
|
||||||
|
|
||||||
Replacement is destructive and must be narrow.
|
|
||||||
|
|
||||||
`distributor` must never perform broad deletion against a configured destination root.
|
|
||||||
|
|
||||||
Replacement may occur only at a resolved destination bundle path when:
|
|
||||||
|
|
||||||
- a valid `.distributor.json` exists at that destination bundle path; and
|
|
||||||
- the state identifies the path as distributor-managed; and
|
|
||||||
- the replacement decision follows the destination comparison rules.
|
|
||||||
|
|
||||||
For MVP, replacement should delete only known managed outputs where practical:
|
|
||||||
|
|
||||||
- files listed in existing `.distributor.json.outputs`;
|
|
||||||
- existing `.distributor.json`;
|
|
||||||
- empty directories created by those files, where applicable for filesystem-like backends.
|
|
||||||
|
|
||||||
For S3, replacement should delete only objects under the destination bundle prefix that are listed in `.distributor.json.outputs` plus `.distributor.json`, unless a later managed-prefix deletion policy is explicitly implemented.
|
|
||||||
|
|
||||||
Before any write, planned destination output paths must be checked for collisions. For example, if `report.md` is copied as a source artifact and Markdown transformation would also generate `report.html`, but `report.html` is already a source artifact or another generated output, planning must fail before writing.
|
|
||||||
|
|
||||||
## Transform Output Contract
|
|
||||||
|
|
||||||
Source files are canonical. Generated files are derived publication artifacts.
|
|
||||||
|
|
||||||
For MVP, the only supported transform is Markdown to HTML.
|
|
||||||
|
|
||||||
Recommended MVP behavior:
|
|
||||||
|
|
||||||
- Transform is configured per destination.
|
|
||||||
- Markdown source files are files listed in `manifest.json` with `.md` extension.
|
|
||||||
- Generated HTML files are sidecars by default.
|
|
||||||
- `report.md` generates `report.html`.
|
|
||||||
- Source bundles are never mutated.
|
|
||||||
- Generated outputs are recorded in `.distributor.json.outputs`.
|
|
||||||
- Raw HTML embedded in Markdown is escaped or disabled by default for the MVP to keep generated output deterministic and conservative.
|
|
||||||
|
|
||||||
Future versions may add templates, `index.html`, CSS assets, email-safe HTML, and per-file transform selection.
|
|
||||||
|
|
||||||
## Destination Output Contract
|
|
||||||
|
|
||||||
Each destination chooses which categories of files it receives.
|
|
||||||
|
|
||||||
MVP categories:
|
|
||||||
|
|
||||||
- `source`: copied source artifacts listed in `manifest.json`.
|
|
||||||
- `html`: generated HTML derived from Markdown source artifacts.
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
|
|
||||||
- Markdown archive: `source: true`, `html: false`.
|
|
||||||
- Static HTML site: `source: false`, `html: true`.
|
|
||||||
- Full mirror: `source: true`, `html: true`.
|
|
||||||
|
|
||||||
Every file written to the destination must be represented in `.distributor.json.outputs`.
|
|
||||||
|
|
||||||
## Atomicity and Partial Failure
|
|
||||||
|
|
||||||
The MVP should prefer staging and promotion where backend semantics permit it.
|
|
||||||
|
|
||||||
Minimum behavior:
|
|
||||||
|
|
||||||
- Validate source before writing destination files.
|
|
||||||
- Do not write `.distributor.json` until all configured destination outputs are successfully written.
|
|
||||||
- If publication fails before `.distributor.json` is written, the destination must not be treated as successfully published on a later run.
|
|
||||||
- Local publication must use staging or equivalent cleanup behavior so a failed write does not leave a confusing unmanaged destination bundle path.
|
|
||||||
- Later cleanup may remove orphaned files, but MVP correctness should rely on `.distributor.json` as the success marker.
|
|
||||||
|
|
||||||
## Example Source Bundle
|
|
||||||
|
|
||||||
```text
|
|
||||||
weather/daily/brentwood/2026-05-30/
|
|
||||||
manifest.json
|
|
||||||
report.md
|
|
||||||
summary.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
Example manifest:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema_version": 1,
|
|
||||||
"id": "weather.daily.brentwood.2026-05-30",
|
|
||||||
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
|
||||||
"created": "2026-05-30T11:10:00Z",
|
|
||||||
"files": [
|
|
||||||
{
|
|
||||||
"path": "report.md",
|
|
||||||
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
|
||||||
"size": 12345
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"path": "summary.txt",
|
|
||||||
"sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
|
|
||||||
"size": 234
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Example Destination Bundle: HTML Only
|
|
||||||
|
|
||||||
```text
|
|
||||||
weather/daily/brentwood/2026-05-30/
|
|
||||||
report.html
|
|
||||||
.distributor.json
|
|
||||||
```
|
|
||||||
|
|
||||||
## Example Destination Bundle: Source Archive
|
|
||||||
|
|
||||||
```text
|
|
||||||
weather/daily/brentwood/2026-05-30/
|
|
||||||
report.md
|
|
||||||
summary.txt
|
|
||||||
.distributor.json
|
|
||||||
```
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
1. Define Go structs for source manifest and destination state.
|
|
||||||
2. Implement source path validation.
|
|
||||||
3. Implement per-file SHA256 and size validation.
|
|
||||||
4. Implement canonical bundle digest validation.
|
|
||||||
5. Implement source bundle discovery beneath a source root.
|
|
||||||
6. Implement `.distributor.json` parsing and validation.
|
|
||||||
7. Implement destination comparison rules.
|
|
||||||
8. Implement replacement safety checks.
|
|
||||||
9. Add fixture bundles for valid, invalid, duplicate path, older, newer, same, and conflict scenarios.
|
|
||||||
10. Add reference canonical digest fixtures.
|
|
||||||
11. Use the contract layer from the publish pipeline and backend adapters.
|
|
||||||
@@ -1,468 +0,0 @@
|
|||||||
# Documentation Roadmap
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
This roadmap defines a documentation-only refresh plan for `distributor` after the local MVP and cleanup roadmap implementation.
|
|
||||||
|
|
||||||
The goal is to make current-behavior documentation concise, accurate, and compliant with `docs/policy/documentation.md` while keeping planned, aspirational, or unimplemented behavior under `docs/roadmap/`.
|
|
||||||
|
|
||||||
This file is written for an LLM coding agent that will implement the documentation refresh in stages. It does not itself rewrite current user, policy, internal, or example documentation.
|
|
||||||
|
|
||||||
## Repository Documentation Inventory
|
|
||||||
|
|
||||||
The current documentation and examples reviewed are:
|
|
||||||
|
|
||||||
| Path | Current status | Notes |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `README.md` | Keep and lightly update | Short, accurate orientation page with a runnable local example command. |
|
|
||||||
| `docs/cli.md` | Keep and update against CLI tests | Covers `version`, `run`, `validate`, and `inspect`; should be checked against command help and parser tests. |
|
|
||||||
| `docs/config.md` | Keep and tighten | Current config reference documents local execution and also accepted SSH/S3 config fields. It must clearly distinguish config validation from executable backend support. |
|
|
||||||
| `docs/operations.md` | Keep and expand slightly | Covers local workflow, destination state, retry behavior, cleanup, fan-out failure handling, and caveats. |
|
|
||||||
| `docs/troubleshooting.md` | Added | Covers recurring local MVP failure modes. |
|
|
||||||
| `docs/policy/architecture.md` | Keep and clarify where needed | Development policy is broad and includes future adapter direction. Wording should not imply SSH/S3 adapters currently exist. |
|
|
||||||
| `docs/policy/development.md` | Updated | Contains contributor and agent workflow guidance. |
|
|
||||||
| `docs/policy/documentation.md` | Keep | Canonical documentation policy. No change required unless the policy itself changes. |
|
|
||||||
| `docs/internal/bundle.md` | Keep and verify | Describes implemented bundle parsing, discovery, validation, and digest semantics. |
|
|
||||||
| `docs/internal/notify.md` | Keep and verify | Accurately states current no-op notification behavior. |
|
|
||||||
| `docs/internal/publish.md` | Keep and verify | Describes planning, execution, replacement, safety, and current local scope. |
|
|
||||||
| `docs/internal/state.md` | Keep and verify | Describes implemented `.distributor.json` state and comparison behavior. |
|
|
||||||
| `docs/internal/storage.md` | Keep and verify | Describes storage interface, typed errors, path rules, traversal, and managed deletion. |
|
|
||||||
| `docs/internal/transform.md` | Keep and verify | Describes Markdown-to-HTML sidecar behavior and transform boundaries. |
|
|
||||||
| `docs/internal/app.md` | Added | Covers orchestration, backend factory, transform registry, dry-run, summaries, and notifier handoff. |
|
|
||||||
| `docs/internal/config.md` | Added | Covers config loading, defaults, validation, accepted-but-not-executable backends, and example tests. |
|
|
||||||
| `docs/internal/local-backend.md` | Missing, optional | Local adapter behavior may remain in `docs/internal/storage.md`; create this only if local filesystem safety detail outgrows that doc. |
|
|
||||||
| `docs/integrations/markdown.md` | Added | Documents current Goldmark-backed Markdown rendering behavior. |
|
|
||||||
| `docs/roadmap/audit.md` | Historical roadmap/report | Keep under roadmap unless replaced by a new audit. |
|
|
||||||
| `docs/roadmap/cleanup.md` | Historical or completed roadmap | Keep under roadmap; optionally add completion status in the documentation refresh. |
|
|
||||||
| `docs/roadmap/config.md` | Roadmap | Keep as planning material; avoid linking to it as current config reference. |
|
|
||||||
| `docs/roadmap/contracts.md` | Roadmap | Keep as planning material; current implemented contracts should be summarized in `docs/internal/` and user docs as needed. |
|
|
||||||
| `docs/roadmap/implementation.md` | Roadmap | Keep as implementation history plus future stages; status should be clear. |
|
|
||||||
| `docs/roadmap/packages.md` | Roadmap | Keep as planning material; current package docs belong under `docs/internal/`. |
|
|
||||||
| `docs/roadmap/storage.md` | Roadmap | Keep as planning material; current storage contract belongs in `docs/internal/storage.md`. |
|
|
||||||
| `examples/local-to-local.yml` | Keep | Minimal local config; load-tested. |
|
|
||||||
| `examples/local-publish.yml` | Keep | Runnable local publication example used by README and CLI docs. |
|
|
||||||
| `examples/local-html.yml` | Keep | Runnable local HTML example. |
|
|
||||||
| `examples/fan-out.yml` | Updated | Local-only runnable fan-out example. |
|
|
||||||
| `examples/source-bundle/` | Keep | Copyable valid source bundle fixture for local CLI examples. |
|
|
||||||
|
|
||||||
Implementation source areas inspected for documentation truth:
|
|
||||||
|
|
||||||
- CLI entrypoints: `cmd/distributor`, `internal/cli`.
|
|
||||||
- Application orchestration: `internal/app`.
|
|
||||||
- Config loading/defaults/validation: `internal/config`.
|
|
||||||
- Bundle manifest, discovery, and validation: `internal/bundle`.
|
|
||||||
- Destination state: `internal/state`.
|
|
||||||
- Storage abstraction and local/fake backends: `internal/storage`, `internal/storage/fake`, `internal/adapters/local`.
|
|
||||||
- Publish planning and execution: `internal/publish`.
|
|
||||||
- Transform registry and Markdown renderer: `internal/transform`, `internal/transform/markdown`.
|
|
||||||
- Notification hook: `internal/notify`.
|
|
||||||
- Tests and fixtures: package tests, `internal/testutil`, `examples/`, and `internal/bundle/testdata`.
|
|
||||||
|
|
||||||
Absent areas from earlier planning that should not be documented as implemented:
|
|
||||||
|
|
||||||
- `internal/adapters/ssh`
|
|
||||||
- `internal/adapters/s3`
|
|
||||||
- `internal/stage`
|
|
||||||
- `internal/modules`
|
|
||||||
- `internal/validators`
|
|
||||||
- `internal/artifacts`
|
|
||||||
- `internal/manifest`
|
|
||||||
- `internal/schema`
|
|
||||||
- `internal/report`
|
|
||||||
- `pkg`
|
|
||||||
|
|
||||||
## Policy Compliance Assessment
|
|
||||||
|
|
||||||
Required current-behavior docs now exist for the config-driven, stateful, modular CLI.
|
|
||||||
|
|
||||||
Completed fixes from this refresh:
|
|
||||||
|
|
||||||
- `docs/policy/development.md` contains real workflow guidance.
|
|
||||||
- Non-roadmap docs are scoped to implemented behavior. SSH/S3 execution, force overwrite, and external notification adapters remain described as unavailable unless the corresponding code exists.
|
|
||||||
- `examples/fan-out.yml` is local-only and runnable.
|
|
||||||
|
|
||||||
Completed recommended fixes from this refresh:
|
|
||||||
|
|
||||||
- `docs/troubleshooting.md` covers recurring local MVP failure modes.
|
|
||||||
- `docs/internal/app.md` and `docs/internal/config.md` provide current-behavior internal references.
|
|
||||||
- `docs/integrations/markdown.md` documents current Goldmark-backed rendering behavior.
|
|
||||||
- Historical roadmap status notes were added where useful.
|
|
||||||
|
|
||||||
No broad rewrite is needed for `README.md`, `docs/cli.md`, `docs/config.md`, or `docs/operations.md`. They are close to the implemented local MVP and should be tightened against code and tests.
|
|
||||||
|
|
||||||
## Target Documentation Set
|
|
||||||
|
|
||||||
### `README.md`
|
|
||||||
|
|
||||||
- Audience: users, administrators, operators.
|
|
||||||
- Purpose: short project orientation and fastest useful local command.
|
|
||||||
- Canonical scope: project purpose, elevator pitch, one local quickstart command, links to current docs.
|
|
||||||
- Recommended outline: title, one-sentence description, local example command, links.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/cli`, `internal/app/run.go`, `examples/local-publish.yml`, `docs/cli.md`.
|
|
||||||
- Acceptance criteria: command is executable in the current local MVP; README does not describe SSH, S3, force overwrite, notification adapters, or future roadmap behavior as available.
|
|
||||||
|
|
||||||
### `docs/cli.md`
|
|
||||||
|
|
||||||
- Audience: users, administrators, operators.
|
|
||||||
- Purpose: canonical CLI reference.
|
|
||||||
- Canonical scope: commands, flags, useful workflows, command output expectations, local-only limits.
|
|
||||||
- Recommended outline: shortest useful command, command overview, flag reference, common workflows, diagnostics and recovery commands.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/cli/*.go`, `internal/cli/*_test.go`, `internal/app/validate.go`, `internal/app/inspect.go`, `internal/app/run.go`.
|
|
||||||
- Acceptance criteria: every documented command and flag exists; `validate` and `inspect` are documented as local path commands; `run --dry-run` output is described without over-specifying every line; unsupported remote execution is stated clearly.
|
|
||||||
|
|
||||||
### `docs/config.md`
|
|
||||||
|
|
||||||
- Audience: administrators, operators, advanced users.
|
|
||||||
- Purpose: canonical configuration reference.
|
|
||||||
- Canonical scope: config file path behavior, minimal local config, production-oriented local config, full schema, defaults, validation rules, secrets handling, links to examples.
|
|
||||||
- Recommended outline: config file location, minimal local config, production-oriented local config, reference, defaults, secrets, examples.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/config/config.go`, `internal/config/defaults.go`, `internal/config/load.go`, `internal/config/validate.go`, `internal/config/load_test.go`, `examples/*.yml`.
|
|
||||||
- Acceptance criteria: fields and defaults match code; `KnownFields(true)` behavior is noted where useful; SSH/S3 fields are described as accepted by config validation but not implemented for execution; `on_digest_mismatch: warn` and unmanaged overwrite are not documented as active options.
|
|
||||||
|
|
||||||
### `docs/operations.md`
|
|
||||||
|
|
||||||
- Audience: administrators, operators.
|
|
||||||
- Purpose: operating and recovery notes for the implemented local MVP.
|
|
||||||
- Canonical scope: local workflow, filesystem layout, destination state, dry-run, retry behavior, replacement safety, failed write cleanup, caveats.
|
|
||||||
- Recommended outline: normal workflow, filesystem layout, destination state, dry-run and planning, retry and replacement behavior, failure handling, cleanup behavior, caveats.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/app/run.go`, `internal/publish/plan.go`, `internal/publish/execute.go`, `internal/publish/reconcile.go`, `internal/publish/safety.go`, `internal/state`, `internal/adapters/local`.
|
|
||||||
- Acceptance criteria: describes only local-to-local operation; explains `.distributor.json` as the managed sentinel; distinguishes skip, replace, conflict, and unmanaged destination behavior; does not promise resume, remote storage, force overwrite, or external notifications.
|
|
||||||
|
|
||||||
### `docs/troubleshooting.md`
|
|
||||||
|
|
||||||
- Audience: administrators, operators.
|
|
||||||
- Purpose: symptom-oriented fixes for common local MVP failures.
|
|
||||||
- Canonical scope: implemented failure modes only.
|
|
||||||
- Recommended outline: one entry per symptom with symptom, likely cause, diagnostic step, safe fix, and relevant link.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/config/validate.go`, `internal/bundle/validate.go`, `internal/state/compare.go`, `internal/publish/plan.go`, `internal/publish/output.go`, CLI tests.
|
|
||||||
- Acceptance criteria: entries are actionable and do not suggest unsafe deletion; remote backend failures are described only as unsupported execution; all fixes link to `docs/cli.md`, `docs/config.md`, or `docs/operations.md` where useful.
|
|
||||||
|
|
||||||
### `docs/policy/architecture.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: development principles and architectural invariants.
|
|
||||||
- Canonical scope: project shape, package boundaries, state/persistence philosophy, external integration philosophy, errors/logging, tests, docs, non-goals.
|
|
||||||
- Recommended outline: keep the existing outline.
|
|
||||||
- Source-of-truth repo areas to inspect: full package tree, implemented internal docs, roadmap files for explicitly future work.
|
|
||||||
- Acceptance criteria: still gives long-term architecture direction, but any unimplemented adapter packages or future capabilities are worded as planned/target architecture rather than implemented behavior.
|
|
||||||
|
|
||||||
### `docs/policy/development.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: contributor and agent workflow.
|
|
||||||
- Canonical scope: repo layout, build/test commands, coding conventions, dependency policy, how to add config fields, CLI flags, backends, transforms, examples, and docs.
|
|
||||||
- Recommended outline: repository layout, common commands, coding conventions, dependency policy, adding config fields, adding CLI flags, adding storage backends, adding transforms, updating examples, documentation expectations.
|
|
||||||
- Source-of-truth repo areas to inspect: `go.mod`, `cmd/distributor`, `internal/*`, `examples`, tests, `docs/policy/architecture.md`, `docs/policy/documentation.md`.
|
|
||||||
- Acceptance criteria: no placeholder content remains; commands are real; workflow guidance protects current boundaries; examples and docs update rules match policy.
|
|
||||||
|
|
||||||
### `docs/internal/app.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: implemented orchestration reference.
|
|
||||||
- Canonical scope: `Run`, `Validate`, `Inspect`, backend factory, transform registry, dry-run, per-destination fan-out, failure aggregation, notifier invocation.
|
|
||||||
- Recommended outline: purpose, inputs and outputs, run flow, backend and transform registration, dry-run behavior, failure behavior, notification behavior, tests to inspect, invariants.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/app/*.go`, `internal/app/*_test.go`, `internal/cli/root_test.go`.
|
|
||||||
- Acceptance criteria: documents current local-only backend execution and the no-op default notifier; does not introduce a generic stage framework that does not exist.
|
|
||||||
|
|
||||||
### `docs/internal/config.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: internal config loading/default/validation reference.
|
|
||||||
- Canonical scope: YAML decoding, known-field rejection, defaults, validation error model, backend config shape, publish/transform validation helper, example load tests.
|
|
||||||
- Recommended outline: purpose, inputs and outputs, loading flow, defaults, validation responsibilities, executable support boundary, tests to inspect, invariants.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/config/*.go`, `internal/config/*_test.go`, `docs/config.md`, `examples/*.yml`.
|
|
||||||
- Acceptance criteria: documents that SSH/S3 config validation exists while execution does not; keeps user-facing config reference canonical in `docs/config.md`.
|
|
||||||
|
|
||||||
### `docs/internal/bundle.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: implemented bundle contract and validation reference.
|
|
||||||
- Canonical scope: `manifest.json`, discovery, validation, digest semantics, storage interactions, tests.
|
|
||||||
- Recommended outline: keep current outline and verify against code.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/bundle`, `internal/storage`, `internal/bundle/testdata`, `examples/source-bundle`.
|
|
||||||
- Acceptance criteria: canonical digest, duplicate paths, reserved paths, symlink rejection, RFC3339 parsing, and discovery behavior match implementation.
|
|
||||||
|
|
||||||
### `docs/internal/state.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: destination state and comparison reference.
|
|
||||||
- Canonical scope: `.distributor.json` schema, validation, output metadata, comparison outcomes.
|
|
||||||
- Recommended outline: keep current outline and verify against code.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/state`, `internal/publish/reconcile.go`, `internal/publish/execute.go`.
|
|
||||||
- Acceptance criteria: state schema and comparison outcomes match implemented structs and tests; `distributor_version` is described as optional diagnostic metadata.
|
|
||||||
|
|
||||||
### `docs/internal/storage.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: storage interface and backend safety reference.
|
|
||||||
- Canonical scope: logical paths, IO methods, traversal, `HasAny`, typed errors, managed deletion, local and fake backend behavior.
|
|
||||||
- Recommended outline: keep current outline and add any missing implemented details that matter for callers.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/storage`, `internal/storage/fake`, `internal/adapters/local`.
|
|
||||||
- Acceptance criteria: matches actual `Backend` interface, `WriteOptions`, `DeleteOptions`, `ErrStopWalk`, and `storage.List` helper; does not describe raw recursive delete as available.
|
|
||||||
|
|
||||||
### `docs/internal/publish.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: publish planning and execution reference.
|
|
||||||
- Canonical scope: request inputs, output planning, destination inspection, transfer policy, actions, replacement safety, cleanup on failed writes.
|
|
||||||
- Recommended outline: keep current outline and verify against code.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/publish`, `internal/app/run.go`, `internal/config/defaults.go`.
|
|
||||||
- Acceptance criteria: action names match constants; transfer policy values match code; collision detection and managed deletion behavior are covered.
|
|
||||||
|
|
||||||
### `docs/internal/transform.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: transform registry and Markdown transform reference.
|
|
||||||
- Canonical scope: transform interface, registry, Markdown sidecar output, deterministic output metadata, raw HTML behavior.
|
|
||||||
- Recommended outline: keep current outline; link to integration notes if `docs/integrations/markdown.md` is created.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/transform`, `internal/transform/markdown`, markdown tests.
|
|
||||||
- Acceptance criteria: `.md` to `.html` sidecar naming, skipped non-Markdown files, digest metadata, and source immutability match implementation.
|
|
||||||
|
|
||||||
### `docs/internal/notify.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: notification hook reference.
|
|
||||||
- Canonical scope: interface, no-op notifier, invocation points, non-invocation points.
|
|
||||||
- Recommended outline: keep current outline and add tests to inspect if useful.
|
|
||||||
- Source-of-truth repo areas to inspect: `internal/notify`, `internal/app/run.go`, `internal/app/run_test.go`.
|
|
||||||
- Acceptance criteria: says no external notification adapters or user-facing notification config exist.
|
|
||||||
|
|
||||||
### `docs/integrations/markdown.md`
|
|
||||||
|
|
||||||
- Audience: developers, LLM coding agents.
|
|
||||||
- Purpose: concise external integration note for Markdown rendering.
|
|
||||||
- Canonical scope: Goldmark dependency, renderer defaults used by `markdown.New`, raw HTML behavior as observed in tests, deterministic wrapper template, supported output mode.
|
|
||||||
- Recommended outline: purpose, dependency, behavior used, behavior intentionally not customized, tests to inspect, update rules.
|
|
||||||
- Source-of-truth repo areas to inspect: `go.mod`, `internal/transform/markdown`, markdown tests.
|
|
||||||
- Acceptance criteria: documents only the Markdown renderer behavior actually used; does not claim full CommonMark compatibility beyond Goldmark defaults.
|
|
||||||
|
|
||||||
### `docs/roadmap/*.md`
|
|
||||||
|
|
||||||
- Audience: maintainers, developers, LLM coding agents.
|
|
||||||
- Purpose: future work, historical plans, accepted deferred work, and implementation prompts.
|
|
||||||
- Canonical scope: unimplemented SSH/S3 adapters, force overwrite, notification adapters, future config fields, release readiness, historical cleanup/audit plans.
|
|
||||||
- Recommended outline: add status notes only where useful; avoid rewriting history unless it causes confusion.
|
|
||||||
- Source-of-truth repo areas to inspect: current implementation and each roadmap file.
|
|
||||||
- Acceptance criteria: future work remains under `docs/roadmap/`; completed historical plans are labeled clearly enough that agents do not re-run them blindly.
|
|
||||||
|
|
||||||
## File-by-File Rewrite Guidance
|
|
||||||
|
|
||||||
`README.md`:
|
|
||||||
|
|
||||||
- Cover: purpose, one local quickstart, links.
|
|
||||||
- Avoid: full config schema, internal package details, remote backend promises.
|
|
||||||
- Link to: `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/roadmap/`.
|
|
||||||
- Inspect: `examples/local-publish.yml`, `internal/cli`.
|
|
||||||
- Stale claims to remove: any implication that remote publication is implemented.
|
|
||||||
|
|
||||||
`docs/cli.md`:
|
|
||||||
|
|
||||||
- Cover: real command syntax and current local workflows.
|
|
||||||
- Avoid: roadmap flags such as force overwrite or remote validation.
|
|
||||||
- Link to: `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md` if created.
|
|
||||||
- Inspect: `internal/cli/*_test.go`.
|
|
||||||
- Stale claims to remove: any command or flag not present in `internal/cli`.
|
|
||||||
|
|
||||||
`docs/config.md`:
|
|
||||||
|
|
||||||
- Cover: current config schema and defaults.
|
|
||||||
- Avoid: presenting SSH/S3 as executable backend support.
|
|
||||||
- Link to: examples and operations.
|
|
||||||
- Inspect: `internal/config/defaults.go`, `internal/config/validate.go`, `internal/config/load_test.go`.
|
|
||||||
- Stale claims to remove: `on_digest_mismatch: warn`, unmanaged overwrite, or force replacement as active options.
|
|
||||||
|
|
||||||
`docs/operations.md`:
|
|
||||||
|
|
||||||
- Cover: local destination state, managed cleanup, retry behavior, fan-out failure aggregation.
|
|
||||||
- Avoid: remote storage recovery, force cleanup, external notifier delivery.
|
|
||||||
- Link to: `docs/troubleshooting.md` for symptom-specific fixes.
|
|
||||||
- Inspect: `internal/app/run.go`, `internal/publish/execute.go`, `internal/state`.
|
|
||||||
- Stale claims to remove: any resume or recovery mechanism beyond re-running after safe cleanup.
|
|
||||||
|
|
||||||
`docs/policy/development.md`:
|
|
||||||
|
|
||||||
- Cover: concrete contributor workflow.
|
|
||||||
- Avoid: placeholder text and invented tools.
|
|
||||||
- Link to: architecture and documentation policies.
|
|
||||||
- Inspect: package tree and `go.mod`.
|
|
||||||
- Stale claims to remove: `# Not yet implemented`.
|
|
||||||
|
|
||||||
`docs/internal/*.md`:
|
|
||||||
|
|
||||||
- Cover: implemented component contracts, boundaries, failure behavior, tests to inspect.
|
|
||||||
- Avoid: roadmap package names or future stages as if they exist.
|
|
||||||
- Link to: current user docs only when relevant.
|
|
||||||
- Inspect: package code and tests.
|
|
||||||
- Stale claims to remove: broad future backend behavior outside local/fake abstractions.
|
|
||||||
|
|
||||||
`examples/`:
|
|
||||||
|
|
||||||
- Cover: valid, maintained, copyable examples.
|
|
||||||
- Avoid: examples that look runnable but fail because the backend execution is unsupported.
|
|
||||||
- Link from: README, CLI docs, config docs.
|
|
||||||
- Inspect: `internal/config/load_test.go` and optional CLI smoke commands.
|
|
||||||
- Stale claims to remove: executable remote fan-out examples until remote backends exist.
|
|
||||||
|
|
||||||
## Examples Plan
|
|
||||||
|
|
||||||
Keep these examples:
|
|
||||||
|
|
||||||
- `examples/source-bundle/`: valid source bundle used by local CLI examples.
|
|
||||||
- `examples/local-to-local.yml`: minimal local config. Use in config docs as the minimal schema example.
|
|
||||||
- `examples/local-publish.yml`: primary runnable quickstart config.
|
|
||||||
- `examples/local-html.yml`: runnable Markdown-to-HTML example.
|
|
||||||
|
|
||||||
Revise `examples/fan-out.yml` in the documentation refresh:
|
|
||||||
|
|
||||||
- Preferred option: replace it with a local-only fan-out example using two local destinations, such as one source archive destination and one HTML destination under `workspace/`.
|
|
||||||
- Alternative option: move the SSH/S3 fan-out material under a roadmap file and remove it from `examples/`.
|
|
||||||
- Do not keep a non-roadmap example that appears copyable for execution but uses unsupported SSH/S3 execution.
|
|
||||||
|
|
||||||
Future example tests should continue loading every YAML file under `examples/`. If `examples/fan-out.yml` becomes local-only, add or update a CLI/app test that exercises local fan-out behavior or rely on existing fan-out tests if they cover equivalent behavior.
|
|
||||||
|
|
||||||
## Internal Documentation Plan
|
|
||||||
|
|
||||||
Update existing internal docs only after checking package code and tests.
|
|
||||||
|
|
||||||
Create these internal docs:
|
|
||||||
|
|
||||||
- `docs/internal/app.md`: orchestration, backend factory, transform registry, dry-run behavior, fan-out, failure aggregation, notifier invocation.
|
|
||||||
- `docs/internal/config.md`: YAML loading, defaults, validation, accepted config fields, unsupported execution boundary, example load tests.
|
|
||||||
|
|
||||||
Keep and verify these internal docs:
|
|
||||||
|
|
||||||
- `docs/internal/bundle.md`
|
|
||||||
- `docs/internal/state.md`
|
|
||||||
- `docs/internal/storage.md`
|
|
||||||
- `docs/internal/publish.md`
|
|
||||||
- `docs/internal/transform.md`
|
|
||||||
- `docs/internal/notify.md`
|
|
||||||
|
|
||||||
Defer these internal docs unless the corresponding implementation grows:
|
|
||||||
|
|
||||||
- `docs/internal/local-backend.md`: create only if local adapter details become too long for `docs/internal/storage.md`.
|
|
||||||
- `docs/internal/logging.md`: defer until logging has behavior beyond placeholders.
|
|
||||||
- `docs/internal/testutil.md`: defer unless test fixture helpers become a stable contributor-facing contract.
|
|
||||||
|
|
||||||
Do not create docs for packages or directories that do not exist.
|
|
||||||
|
|
||||||
## Integration Documentation Plan
|
|
||||||
|
|
||||||
No external service integration docs should be created for SSH, S3, or notification services until those integrations are implemented.
|
|
||||||
|
|
||||||
Recommended current integration doc:
|
|
||||||
|
|
||||||
- `docs/integrations/markdown.md`
|
|
||||||
|
|
||||||
This should document Goldmark usage because Markdown rendering is an implemented external file-format integration with externally visible output. Keep it concise and limited to:
|
|
||||||
|
|
||||||
- dependency and version source: `go.mod`;
|
|
||||||
- renderer construction: `goldmark.New()`;
|
|
||||||
- sidecar output path behavior owned by `internal/transform/markdown`;
|
|
||||||
- raw HTML behavior covered by tests;
|
|
||||||
- wrapper template behavior;
|
|
||||||
- tests to inspect before changing renderer behavior.
|
|
||||||
|
|
||||||
Do not create separate YAML integration docs unless configuration parsing behavior outgrows `docs/config.md` and `docs/internal/config.md`.
|
|
||||||
|
|
||||||
## Recommended Implementation Sequence
|
|
||||||
|
|
||||||
### Stage 1: Establish Documentation Status and Contributor Policy
|
|
||||||
|
|
||||||
- Goal: remove the required policy placeholder and classify active versus historical documentation.
|
|
||||||
- Files to create/update/delete/move: update `docs/policy/development.md`; optionally add short status notes to `docs/roadmap/audit.md` and `docs/roadmap/cleanup.md`.
|
|
||||||
- Repo areas to inspect: `go.mod`, `cmd/distributor`, `internal/*`, `examples`, `docs/policy/architecture.md`, `docs/policy/documentation.md`.
|
|
||||||
- Acceptance criteria: `docs/policy/development.md` contains real workflow guidance; no active policy doc says "Not yet implemented"; roadmap status notes do not change current behavior docs.
|
|
||||||
- Suggested validation commands: `rg -n "Not yet implemented" docs/policy README.md docs/*.md docs/internal`; `git diff -- docs/policy/development.md docs/roadmap/audit.md docs/roadmap/cleanup.md`.
|
|
||||||
- One prompt: yes.
|
|
||||||
|
|
||||||
### Stage 2: Tighten Current User and Operator Docs
|
|
||||||
|
|
||||||
- Goal: make README, CLI, config, and operations docs exactly match the local MVP.
|
|
||||||
- Files to create/update/delete/move: update `README.md`, `docs/cli.md`, `docs/config.md`, `docs/operations.md`; create `docs/troubleshooting.md`.
|
|
||||||
- Repo areas to inspect: `internal/cli`, `internal/app`, `internal/config`, `internal/publish`, `internal/state`, `examples`.
|
|
||||||
- Acceptance criteria: user docs describe local execution, local validation/inspection, current config schema, current defaults, current state/retry behavior, and clear unsupported remote execution boundaries.
|
|
||||||
- Suggested validation commands: `rg -n "force|allow_unmanaged|on_digest_mismatch: warn|warn" README.md docs/cli.md docs/config.md docs/operations.md docs/troubleshooting.md`; `rg -n "ssh|s3|remote|notification" README.md docs/cli.md docs/config.md docs/operations.md docs/troubleshooting.md`.
|
|
||||||
- One prompt: yes.
|
|
||||||
|
|
||||||
### Stage 3: Make Examples Fully Runnable or Clearly Roadmap-Only
|
|
||||||
|
|
||||||
- Goal: ensure `examples/` contains implemented, copyable examples only.
|
|
||||||
- Files to create/update/delete/move: update or replace `examples/fan-out.yml`; update links in `README.md`, `docs/cli.md`, and `docs/config.md` if needed; move remote fan-out material to a roadmap section if preserving it is useful.
|
|
||||||
- Repo areas to inspect: `internal/config/load_test.go`, `internal/app/run_test.go`, `examples`.
|
|
||||||
- Acceptance criteria: every example under `examples/` is valid current config and does not rely on unsupported remote execution; primary examples remain load-tested.
|
|
||||||
- Suggested validation commands: `go test ./internal/config`; optional `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`; optional `go run ./cmd/distributor run --config examples/local-html.yml --dry-run`.
|
|
||||||
- One prompt: yes.
|
|
||||||
|
|
||||||
### Stage 4: Complete Internal Component Docs
|
|
||||||
|
|
||||||
- Goal: give future agents a current-behavior internal map before remote backends are added.
|
|
||||||
- Files to create/update/delete/move: create `docs/internal/app.md` and `docs/internal/config.md`; update existing `docs/internal/*.md` as needed.
|
|
||||||
- Repo areas to inspect: `internal/app`, `internal/config`, `internal/bundle`, `internal/state`, `internal/storage`, `internal/storage/fake`, `internal/adapters/local`, `internal/publish`, `internal/transform`, `internal/notify`, package tests.
|
|
||||||
- Acceptance criteria: every major implemented component has a concise doc with purpose, inputs/outputs, boundaries, failure behavior, tests to inspect, and invariants; no internal doc describes absent SSH/S3 adapters as implemented.
|
|
||||||
- Suggested validation commands: `rg -n "internal/adapters/ssh|internal/adapters/s3|not implemented|future" docs/internal`; `git diff -- docs/internal`.
|
|
||||||
- One prompt: yes.
|
|
||||||
|
|
||||||
### Stage 5: Add Markdown Integration Notes
|
|
||||||
|
|
||||||
- Goal: document the one implemented external file-format integration where behavior matters.
|
|
||||||
- Files to create/update/delete/move: create `docs/integrations/markdown.md`; optionally link from `docs/internal/transform.md`.
|
|
||||||
- Repo areas to inspect: `go.mod`, `internal/transform/markdown`, markdown tests.
|
|
||||||
- Acceptance criteria: the doc is concise, version-aware through `go.mod`, and limited to current Goldmark usage and observed renderer behavior.
|
|
||||||
- Suggested validation commands: `go test ./internal/transform/markdown`; `rg -n "Goldmark|markdown" docs/integrations docs/internal/transform.md`.
|
|
||||||
- One prompt: yes.
|
|
||||||
|
|
||||||
### Stage 6: Final Documentation Consistency Sweep
|
|
||||||
|
|
||||||
- Goal: catch stale links, stale roadmap references, and unimplemented claims outside roadmap.
|
|
||||||
- Files to create/update/delete/move: any docs touched in earlier stages.
|
|
||||||
- Repo areas to inspect: all docs and examples.
|
|
||||||
- Acceptance criteria: documentation is current, concise, link-consistent, and policy-compliant.
|
|
||||||
- Suggested validation commands: `go test ./...`; `rg -n "go-application-template|maximumdirect.net|docs/architecture.md|docs/documentation.md" README.md docs examples`; `rg -n "allow_unmanaged_overwrite|on_digest_mismatch: warn" README.md docs examples`; `rg -n "not implemented|future|planned|roadmap|ssh|s3|remote" README.md docs/*.md docs/internal docs/policy examples`.
|
|
||||||
- One prompt: yes, after the prior stages are complete.
|
|
||||||
|
|
||||||
## Validation Plan
|
|
||||||
|
|
||||||
Run Go tests when examples, CLI docs, config docs, or behavior-linked docs change:
|
|
||||||
|
|
||||||
- `go test ./internal/config`: validates example config loading and config semantics.
|
|
||||||
- `go test ./internal/cli ./internal/app`: validates documented command behavior and local run workflows.
|
|
||||||
- `go test ./internal/transform/markdown`: validates documented Markdown rendering behavior.
|
|
||||||
- `go test ./...`: final broad verification after documentation and example changes.
|
|
||||||
|
|
||||||
Run grep checks:
|
|
||||||
|
|
||||||
- `rg -n "Not yet implemented" docs/policy README.md docs/*.md docs/internal`
|
|
||||||
- `rg -n "go-application-template|docs/architecture.md|docs/documentation.md" README.md docs examples`
|
|
||||||
- `rg -n "allow_unmanaged_overwrite|on_digest_mismatch: warn" README.md docs examples`
|
|
||||||
- `rg -n "force overwrite|--force|remote backends are implemented|notification adapters" README.md docs/*.md docs/internal docs/policy examples`
|
|
||||||
- `rg -n "ssh|s3|remote" README.md docs/*.md docs/internal docs/policy examples`
|
|
||||||
|
|
||||||
The final SSH/S3/remote grep is not expected to return zero results. Manually review every result and confirm it is either:
|
|
||||||
|
|
||||||
- under `docs/roadmap/`;
|
|
||||||
- a clearly stated unsupported-execution boundary;
|
|
||||||
- a config-validation reference that does not imply executable support; or
|
|
||||||
- an architecture policy statement phrased as future/target direction rather than implemented behavior.
|
|
||||||
|
|
||||||
Manual review checklist:
|
|
||||||
|
|
||||||
- README remains short.
|
|
||||||
- `docs/config.md` is the only current-behavior config reference.
|
|
||||||
- `docs/cli.md` is the only current-behavior CLI reference.
|
|
||||||
- `docs/operations.md` covers state and recovery without unsafe deletion advice.
|
|
||||||
- `docs/troubleshooting.md` is symptom-oriented and links to canonical docs.
|
|
||||||
- Internal docs point to tests before changing behavior.
|
|
||||||
- Examples are copyable and free of secrets.
|
|
||||||
- Future work remains under `docs/roadmap/`.
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
No blocking questions remain before implementing this documentation refresh.
|
|
||||||
|
|
||||||
The only non-blocking choice is how to handle `examples/fan-out.yml`:
|
|
||||||
|
|
||||||
- Preferred: convert it to a local-only fan-out example so `examples/` remains fully runnable.
|
|
||||||
- Acceptable: move the current SSH/S3 fan-out example into roadmap material until remote execution exists.
|
|
||||||
|
|
||||||
Use the preferred option unless a maintainer explicitly wants `examples/` to include config-validated but non-executable examples.
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,526 +0,0 @@
|
|||||||
# Package Layout Roadmap
|
|
||||||
|
|
||||||
This roadmap defines the proposed package layout, boundaries, and implementation responsibilities for the `distributor` MVP.
|
|
||||||
|
|
||||||
`distributor` is expected to be a domain-agnostic bundle publisher. Producer applications emit source bundles containing `manifest.json`; `distributor` validates those bundles and publishes selected source and generated artifacts to one or more configured destinations.
|
|
||||||
|
|
||||||
This document is roadmap material. It describes the intended package design before implementation and should move into `docs/internal/` only after corresponding behavior exists.
|
|
||||||
|
|
||||||
## Accepted Package Layout
|
|
||||||
|
|
||||||
```text
|
|
||||||
cmd/distributor/
|
|
||||||
main.go
|
|
||||||
|
|
||||||
internal/app/
|
|
||||||
app.go
|
|
||||||
run.go
|
|
||||||
pipeline.go
|
|
||||||
|
|
||||||
internal/cli/
|
|
||||||
root.go
|
|
||||||
run.go
|
|
||||||
validate.go
|
|
||||||
inspect.go
|
|
||||||
|
|
||||||
internal/config/
|
|
||||||
config.go
|
|
||||||
defaults.go
|
|
||||||
load.go
|
|
||||||
validate.go
|
|
||||||
|
|
||||||
internal/bundle/
|
|
||||||
manifest.go
|
|
||||||
digest.go
|
|
||||||
validate.go
|
|
||||||
discover.go
|
|
||||||
|
|
||||||
internal/state/
|
|
||||||
distributor.go
|
|
||||||
compare.go
|
|
||||||
validate.go
|
|
||||||
|
|
||||||
internal/storage/
|
|
||||||
backend.go
|
|
||||||
registry.go
|
|
||||||
path.go
|
|
||||||
errors.go
|
|
||||||
|
|
||||||
internal/storage/fake/
|
|
||||||
backend.go
|
|
||||||
|
|
||||||
internal/adapters/local/
|
|
||||||
backend.go
|
|
||||||
|
|
||||||
internal/adapters/ssh/
|
|
||||||
backend.go
|
|
||||||
config.go
|
|
||||||
|
|
||||||
internal/adapters/s3/
|
|
||||||
backend.go
|
|
||||||
config.go
|
|
||||||
|
|
||||||
internal/transform/
|
|
||||||
transform.go
|
|
||||||
registry.go
|
|
||||||
plan.go
|
|
||||||
|
|
||||||
internal/transform/markdown/
|
|
||||||
markdown.go
|
|
||||||
template.go
|
|
||||||
|
|
||||||
internal/publish/
|
|
||||||
plan.go
|
|
||||||
execute.go
|
|
||||||
output.go
|
|
||||||
reconcile.go
|
|
||||||
safety.go
|
|
||||||
|
|
||||||
internal/notify/
|
|
||||||
notify.go
|
|
||||||
noop.go
|
|
||||||
|
|
||||||
internal/logging/
|
|
||||||
logging.go
|
|
||||||
```
|
|
||||||
|
|
||||||
## Package Responsibilities
|
|
||||||
|
|
||||||
### `cmd/distributor`
|
|
||||||
|
|
||||||
Application entrypoint only.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- call CLI execution;
|
|
||||||
- translate process exit status;
|
|
||||||
- avoid business logic.
|
|
||||||
|
|
||||||
Non-responsibilities:
|
|
||||||
|
|
||||||
- config loading;
|
|
||||||
- backend construction;
|
|
||||||
- bundle validation;
|
|
||||||
- publish decisions.
|
|
||||||
|
|
||||||
### `internal/cli`
|
|
||||||
|
|
||||||
CLI command definitions, flags, argument parsing, and command wiring.
|
|
||||||
|
|
||||||
Expected MVP commands:
|
|
||||||
|
|
||||||
- `distributor run` — run configured pipelines.
|
|
||||||
- `distributor run --dry-run` — plan without modifying destinations.
|
|
||||||
- `distributor run --pipeline <id>` — run one configured pipeline.
|
|
||||||
- `distributor validate <path>` — validate a source bundle or source tree where feasible.
|
|
||||||
- `distributor inspect <path>` — inspect a bundle or destination state where feasible.
|
|
||||||
|
|
||||||
Boundaries:
|
|
||||||
|
|
||||||
- CLI should call `internal/app` use cases.
|
|
||||||
- CLI should not parse manifests directly except through application APIs.
|
|
||||||
- CLI should not import backend adapter implementation details unless only for registration side effects.
|
|
||||||
|
|
||||||
### `internal/app`
|
|
||||||
|
|
||||||
Application orchestration and top-level use cases.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- load and validate configuration;
|
|
||||||
- construct configured pipelines;
|
|
||||||
- build source and destination backends through registries;
|
|
||||||
- orchestrate discovery, validation, planning, publishing, and notification;
|
|
||||||
- coordinate dry-run output;
|
|
||||||
- run destination fan-out deterministically and sequentially;
|
|
||||||
- aggregate destination outcomes into run-level failure behavior.
|
|
||||||
|
|
||||||
Core orchestration shape:
|
|
||||||
|
|
||||||
```text
|
|
||||||
for each selected pipeline:
|
|
||||||
open source backend
|
|
||||||
discover source bundles
|
|
||||||
for each source bundle:
|
|
||||||
validate source manifest and digest
|
|
||||||
for each destination:
|
|
||||||
inspect .distributor.json
|
|
||||||
build publish plan
|
|
||||||
transform as required by that destination
|
|
||||||
execute publish plan unless dry-run
|
|
||||||
run noop notifier after actual publication or replacement
|
|
||||||
```
|
|
||||||
|
|
||||||
Boundaries:
|
|
||||||
|
|
||||||
- `internal/app` composes packages but should not contain backend-specific logic.
|
|
||||||
- Publish decisions should live in `internal/publish`, not inline in orchestration.
|
|
||||||
- Destination state comparison should live in `internal/state` or `internal/publish`, not CLI code.
|
|
||||||
|
|
||||||
### `internal/config`
|
|
||||||
|
|
||||||
Configuration structs, defaults, loading, precedence, and validation.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- load `/usr/local/etc/distributor/config.yml` by default;
|
|
||||||
- support `--config` override;
|
|
||||||
- apply defaults;
|
|
||||||
- validate required fields;
|
|
||||||
- validate pipeline ids and destination ids;
|
|
||||||
- validate backend-specific config shapes;
|
|
||||||
- validate transform and publish policy combinations.
|
|
||||||
|
|
||||||
MVP config model:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
pipelines:
|
|
||||||
- id: weather-daily
|
|
||||||
source:
|
|
||||||
backend: local
|
|
||||||
path: /var/spool/distributor/weather
|
|
||||||
validation:
|
|
||||||
on_digest_mismatch: fail
|
|
||||||
destinations:
|
|
||||||
- id: markdown-archive
|
|
||||||
backend: s3
|
|
||||||
endpoint: https://s3.example.com
|
|
||||||
bucket: reports
|
|
||||||
prefix: weather/archive
|
|
||||||
region: us-east-1
|
|
||||||
force_path_style: true
|
|
||||||
publish:
|
|
||||||
source: true
|
|
||||||
html: false
|
|
||||||
transfer:
|
|
||||||
on_destination_same: skip
|
|
||||||
on_destination_older: replace
|
|
||||||
on_destination_newer: skip
|
|
||||||
on_conflict: fail
|
|
||||||
- id: static-site
|
|
||||||
backend: ssh
|
|
||||||
uri: ssh://deploy@example.com:22
|
|
||||||
path: /srv/www/weather
|
|
||||||
publish:
|
|
||||||
source: false
|
|
||||||
html: true
|
|
||||||
transform:
|
|
||||||
markdown_to_html:
|
|
||||||
enabled: true
|
|
||||||
mode: sidecar
|
|
||||||
```
|
|
||||||
|
|
||||||
Configuration principles:
|
|
||||||
|
|
||||||
- one source per pipeline;
|
|
||||||
- one or more destinations per pipeline;
|
|
||||||
- transforms are destination-specific;
|
|
||||||
- publish policy is destination-specific;
|
|
||||||
- secrets should use environment variables, secret files, SSH agent, or standard credential mechanisms rather than raw YAML values.
|
|
||||||
|
|
||||||
### `internal/bundle`
|
|
||||||
|
|
||||||
Source bundle contract and validation.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- parse source `manifest.json`;
|
|
||||||
- represent source manifests and files;
|
|
||||||
- discover bundle roots beneath a configured source root;
|
|
||||||
- validate required manifest fields;
|
|
||||||
- validate RFC3339 `created` values;
|
|
||||||
- validate relative paths;
|
|
||||||
- validate file existence, size, per-file SHA-256, and bundle digest;
|
|
||||||
- expose normalized source bundle models to other packages.
|
|
||||||
|
|
||||||
Core types:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Manifest struct {
|
|
||||||
SchemaVersion int
|
|
||||||
ID string
|
|
||||||
Digest string
|
|
||||||
Created time.Time
|
|
||||||
Files []ManifestFile
|
|
||||||
}
|
|
||||||
|
|
||||||
type ManifestFile struct {
|
|
||||||
Path string
|
|
||||||
SHA256 string
|
|
||||||
Size int64
|
|
||||||
}
|
|
||||||
|
|
||||||
type Bundle struct {
|
|
||||||
RootRelativePath string
|
|
||||||
Manifest Manifest
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Boundaries:
|
|
||||||
|
|
||||||
- `internal/bundle` does not know about `.distributor.json`.
|
|
||||||
- `internal/bundle` does not know about destinations, transforms, or notification.
|
|
||||||
- `internal/bundle` may use the storage abstraction to read source files, but it should not import backend adapter packages.
|
|
||||||
|
|
||||||
### `internal/state`
|
|
||||||
|
|
||||||
Destination state contract for `.distributor.json`.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- parse `.distributor.json`;
|
|
||||||
- validate destination state;
|
|
||||||
- represent copied source outputs and generated outputs;
|
|
||||||
- embed the source manifest used for publication;
|
|
||||||
- compare destination state against a current source manifest;
|
|
||||||
- classify destination state as same, older, newer, conflict, absent, invalid, or unmanaged.
|
|
||||||
|
|
||||||
Core types:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type DistributorState struct {
|
|
||||||
SchemaVersion int
|
|
||||||
DistributorVersion string
|
|
||||||
PipelineID string
|
|
||||||
DestinationID string
|
|
||||||
PublishedAt time.Time
|
|
||||||
Source SourceState
|
|
||||||
Outputs []OutputFile
|
|
||||||
}
|
|
||||||
|
|
||||||
type SourceState struct {
|
|
||||||
Manifest bundle.Manifest
|
|
||||||
}
|
|
||||||
|
|
||||||
type OutputFile struct {
|
|
||||||
Path string
|
|
||||||
Kind string // source | generated
|
|
||||||
SourcePath string
|
|
||||||
Transform string
|
|
||||||
SHA256 string
|
|
||||||
Size int64
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Comparison rules:
|
|
||||||
|
|
||||||
- same source manifest: skip;
|
|
||||||
- same source id, older destination source `created`: replace;
|
|
||||||
- same source id, newer destination source `created`: skip;
|
|
||||||
- same source id, same `created`, different digest: conflict;
|
|
||||||
- different source id: conflict;
|
|
||||||
- pipeline id or destination id mismatch: conflict;
|
|
||||||
- absent state: publish only if safe;
|
|
||||||
- unmanaged non-empty path: fail.
|
|
||||||
|
|
||||||
Boundaries:
|
|
||||||
|
|
||||||
- `internal/state` owns destination state semantics, not publish execution.
|
|
||||||
- `internal/state` should not know about S3, SSH/SFTP, local filesystem details, or Markdown rendering.
|
|
||||||
|
|
||||||
### `internal/storage`
|
|
||||||
|
|
||||||
Backend abstraction and shared storage types.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- define storage backend interfaces;
|
|
||||||
- define object/file metadata types;
|
|
||||||
- define path/prefix helpers;
|
|
||||||
- define common storage errors;
|
|
||||||
- provide backend registry mechanisms;
|
|
||||||
- provide a fake backend for core package tests.
|
|
||||||
|
|
||||||
The detailed storage contract is defined in `docs/roadmap/storage.md`. Core application code should use that storage interface for backend-rooted logical paths, byte and stream IO, metadata, traversal, typed errors, emptiness checks, and managed deletion.
|
|
||||||
|
|
||||||
Destructive APIs should remain narrow. Prefer managed deletion of files recorded in `.distributor.json` instead of broad recursive deletion.
|
|
||||||
|
|
||||||
Boundaries:
|
|
||||||
|
|
||||||
- `internal/storage` should not contain backend implementation details.
|
|
||||||
- Adapter dependencies must not leak through storage interfaces.
|
|
||||||
- The fake backend exists for tests and should not become an application runtime backend.
|
|
||||||
|
|
||||||
### `internal/adapters/local`
|
|
||||||
|
|
||||||
Local filesystem backend.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- implement `storage.Backend` for local paths;
|
|
||||||
- clean and constrain paths;
|
|
||||||
- perform safe reads/writes/listing/deletion;
|
|
||||||
- use atomic writes where practical;
|
|
||||||
- reject unsafe path traversal;
|
|
||||||
- handle symlink policy explicitly.
|
|
||||||
|
|
||||||
Testing expectations:
|
|
||||||
|
|
||||||
- use temporary directories;
|
|
||||||
- verify path traversal rejection;
|
|
||||||
- verify write and delete safety.
|
|
||||||
|
|
||||||
### `internal/adapters/ssh`
|
|
||||||
|
|
||||||
SSH/SFTP backend.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- implement `storage.Backend` over SSH/SFTP;
|
|
||||||
- support `uri` and `path` config;
|
|
||||||
- prefer native SFTP implementation;
|
|
||||||
- use SSH agent, key files, known hosts, or documented auth mechanisms;
|
|
||||||
- avoid raw passwords in config unless explicitly designed and documented later;
|
|
||||||
- translate SSH/SFTP errors into storage-level errors.
|
|
||||||
|
|
||||||
Testing expectations:
|
|
||||||
|
|
||||||
- core app tests should use fake backends;
|
|
||||||
- adapter tests may use local test servers or targeted integration tests if practical;
|
|
||||||
- do not require a real production SSH host for normal unit tests.
|
|
||||||
|
|
||||||
### `internal/adapters/s3`
|
|
||||||
|
|
||||||
S3-compatible object storage backend.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- implement `storage.Backend` over S3-compatible object storage;
|
|
||||||
- support endpoint, bucket, prefix, region, and force-path-style configuration;
|
|
||||||
- support standard credential mechanisms or explicit environment-variable references;
|
|
||||||
- treat S3 as an object tree, not a filesystem;
|
|
||||||
- set reasonable content types where practical;
|
|
||||||
- guard against prefix/root deletion mistakes.
|
|
||||||
|
|
||||||
Testing expectations:
|
|
||||||
|
|
||||||
- core app tests should use fake backends;
|
|
||||||
- adapter behavior may be tested through mocks, local S3-compatible services, or narrow integration tests;
|
|
||||||
- config examples should avoid real secrets.
|
|
||||||
|
|
||||||
### `internal/transform`
|
|
||||||
|
|
||||||
Transform interfaces, registry, and transform planning.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- define transform interfaces;
|
|
||||||
- register available transforms;
|
|
||||||
- represent transform requests and outputs;
|
|
||||||
- keep transform execution independent of destination backend details.
|
|
||||||
|
|
||||||
Boundaries:
|
|
||||||
|
|
||||||
- transforms operate on source bundle content and destination transform config;
|
|
||||||
- transforms do not publish outputs;
|
|
||||||
- transforms do not mutate source bundles;
|
|
||||||
- transforms should return generated output metadata for `.distributor.json`.
|
|
||||||
|
|
||||||
### `internal/transform/markdown`
|
|
||||||
|
|
||||||
Markdown-to-HTML implementation.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- render listed Markdown files to HTML;
|
|
||||||
- support MVP sidecar behavior, such as `report.md` -> `report.html`;
|
|
||||||
- record generated output path, source path, transform name, SHA-256, and size;
|
|
||||||
- optionally use embedded templates if needed.
|
|
||||||
|
|
||||||
MVP scope:
|
|
||||||
|
|
||||||
- Markdown to HTML only;
|
|
||||||
- no PDF generation;
|
|
||||||
- no email-specific HTML;
|
|
||||||
- no complex theming unless required for basic output correctness.
|
|
||||||
|
|
||||||
### `internal/publish`
|
|
||||||
|
|
||||||
Destination planning, reconciliation, safety checks, and publish execution.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- inspect destination state;
|
|
||||||
- plan destination action;
|
|
||||||
- enforce destination conflict rules;
|
|
||||||
- enforce destructive-operation safety rules;
|
|
||||||
- detect output path collisions before writing;
|
|
||||||
- combine source files and transform outputs according to destination publish policy;
|
|
||||||
- write destination outputs;
|
|
||||||
- write `.distributor.json`;
|
|
||||||
- use staging or equivalent cleanup behavior where practical;
|
|
||||||
- support dry-run planning;
|
|
||||||
- report skipped, replaced, failed, and published actions.
|
|
||||||
|
|
||||||
Action model:
|
|
||||||
|
|
||||||
```text
|
|
||||||
publish
|
|
||||||
replace
|
|
||||||
skip_same
|
|
||||||
skip_destination_newer
|
|
||||||
fail_conflict
|
|
||||||
fail_unmanaged
|
|
||||||
```
|
|
||||||
|
|
||||||
Boundaries:
|
|
||||||
|
|
||||||
- publish logic should not parse CLI flags;
|
|
||||||
- publish logic should not know adapter implementation details;
|
|
||||||
- publish logic should use `internal/state` for destination state semantics;
|
|
||||||
- publish logic should use `internal/storage` interfaces for IO.
|
|
||||||
|
|
||||||
### `internal/notify`
|
|
||||||
|
|
||||||
Notification stage abstraction.
|
|
||||||
|
|
||||||
MVP responsibilities:
|
|
||||||
|
|
||||||
- define notifier interface;
|
|
||||||
- implement no-op notifier;
|
|
||||||
- preserve future extension point for email, ntfy, Gotify, RSS update hooks, or other notification channels.
|
|
||||||
|
|
||||||
Future notification rules:
|
|
||||||
|
|
||||||
- notify only after successful publication to the relevant destination or destinations;
|
|
||||||
- notification must be idempotent with respect to source id, digest, pipeline id, and destination id where applicable;
|
|
||||||
- notification should not run for skipped or failed publications unless explicitly configured.
|
|
||||||
|
|
||||||
### `internal/logging`
|
|
||||||
|
|
||||||
Logging setup and helpers.
|
|
||||||
|
|
||||||
Responsibilities:
|
|
||||||
|
|
||||||
- centralize structured logging setup;
|
|
||||||
- ensure logs omit secrets;
|
|
||||||
- provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason.
|
|
||||||
|
|
||||||
## Deferred Ideas
|
|
||||||
|
|
||||||
The following are intentionally out of MVP unless separately accepted in a later roadmap:
|
|
||||||
|
|
||||||
- email, ntfy, Gotify, or other real notification adapters;
|
|
||||||
- RSS/Atom feed generation;
|
|
||||||
- PDF generation;
|
|
||||||
- web UI;
|
|
||||||
- full-text search;
|
|
||||||
- dynamic plugin loading;
|
|
||||||
- arbitrary transform chains;
|
|
||||||
- workflow DAGs;
|
|
||||||
- producer execution;
|
|
||||||
- complex templating/theming;
|
|
||||||
- bidirectional sync;
|
|
||||||
- backup semantics.
|
|
||||||
|
|
||||||
## Key Invariants
|
|
||||||
|
|
||||||
- Producer apps own source bundle creation.
|
|
||||||
- `distributor` owns destination publication state.
|
|
||||||
- Source `manifest.json` is not copied as destination state.
|
|
||||||
- Destination `.distributor.json` is the managed sentinel.
|
|
||||||
- One pipeline has one source and one or more destinations.
|
|
||||||
- Transform and publish policy are destination-specific.
|
|
||||||
- Source files are canonical; HTML is derived.
|
|
||||||
- Destructive replacement is allowed only inside managed destination bundle paths.
|
|
||||||
- Core logic must be testable without real S3, SSH, or remote services.
|
|
||||||
@@ -1,265 +0,0 @@
|
|||||||
# Storage Interface Roadmap
|
|
||||||
|
|
||||||
This roadmap defines the planned `internal/storage` contract for the `distributor` MVP. The goal is to give bundle validation, destination state inspection, publish planning, and backend adapters one consistent IO boundary without leaking local filesystem, SSH/SFTP, or S3-specific behavior into core packages.
|
|
||||||
|
|
||||||
## Purpose and Invariants
|
|
||||||
|
|
||||||
The storage layer is responsible for safe, backend-rooted access to files, objects, prefixes, and destination bundle paths.
|
|
||||||
|
|
||||||
Core invariants:
|
|
||||||
|
|
||||||
- Backends are opened at configured roots.
|
|
||||||
- Core packages operate on backend-rooted logical paths, not absolute filesystem paths or raw object keys.
|
|
||||||
- Backend adapters translate native storage behavior into common storage entries and typed errors.
|
|
||||||
- Destructive operations remain narrow and managed.
|
|
||||||
- Staging or atomic write behavior belongs behind the storage interface where practical.
|
|
||||||
- The fake backend exists for tests only and must not be registered as a runtime backend.
|
|
||||||
|
|
||||||
## Logical Path Model
|
|
||||||
|
|
||||||
Storage paths are slash-separated logical paths relative to an already configured backend root.
|
|
||||||
|
|
||||||
File paths:
|
|
||||||
|
|
||||||
- must be non-empty;
|
|
||||||
- must be relative;
|
|
||||||
- must be clean;
|
|
||||||
- must not contain `.` or `..` segments;
|
|
||||||
- must not start with `/`;
|
|
||||||
- must not contain backslashes;
|
|
||||||
- must not resolve outside the backend root.
|
|
||||||
|
|
||||||
Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for traversal and destination emptiness checks.
|
|
||||||
|
|
||||||
Prefix matching must preserve logical path boundaries. A prefix of `foo` matches `foo` and entries below `foo/`; it must not match a sibling path such as `foobar`. Backends that map logical paths to object keys must apply the same normalized boundary rule after combining configured backend prefixes with caller-provided logical prefixes.
|
|
||||||
|
|
||||||
Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly.
|
|
||||||
|
|
||||||
## Core Interface Shape
|
|
||||||
|
|
||||||
The MVP should use a hybrid byte and stream interface:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Backend interface {
|
|
||||||
ReadFile(ctx context.Context, path string) ([]byte, error)
|
|
||||||
OpenReader(ctx context.Context, path string) (io.ReadCloser, error)
|
|
||||||
WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) (Entry, error)
|
|
||||||
WriteFrom(ctx context.Context, path string, r io.Reader, opts WriteOptions) (Entry, error)
|
|
||||||
Stat(ctx context.Context, path string) (Entry, error)
|
|
||||||
Walk(ctx context.Context, prefix string, opts WalkOptions, fn WalkFunc) error
|
|
||||||
HasAny(ctx context.Context, prefix string) (bool, error)
|
|
||||||
DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts DeleteOptions) error
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Byte helpers are expected to cover manifests, destination state, small source artifacts, and generated outputs. Stream methods are included from the start for backend flexibility and larger future artifacts.
|
|
||||||
|
|
||||||
Write operations should create required parent directories or prefixes as needed.
|
|
||||||
|
|
||||||
Concrete option and callback types should use this shape:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type WalkOptions struct {
|
|
||||||
Recursive bool
|
|
||||||
Limit int
|
|
||||||
}
|
|
||||||
|
|
||||||
type WalkFunc func(Entry) error
|
|
||||||
|
|
||||||
var ErrStopWalk = errors.New("stop walk")
|
|
||||||
|
|
||||||
type WriteOptions struct {
|
|
||||||
ContentType string
|
|
||||||
Overwrite bool
|
|
||||||
PreferAtomic bool
|
|
||||||
Size int64
|
|
||||||
SizeKnown bool
|
|
||||||
}
|
|
||||||
|
|
||||||
type DeleteOptions struct {
|
|
||||||
IgnoreMissing bool
|
|
||||||
PruneEmptyDirs bool
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`WalkOptions.Limit == 0` means no explicit limit. `SizeKnown` applies primarily to `WriteFrom`; byte writes can infer size from the provided data.
|
|
||||||
|
|
||||||
## Entries and Metadata
|
|
||||||
|
|
||||||
Storage metadata should be represented by an `Entry` model with at least:
|
|
||||||
|
|
||||||
- backend-relative logical path;
|
|
||||||
- entry type;
|
|
||||||
- size, where available.
|
|
||||||
|
|
||||||
Entry types:
|
|
||||||
|
|
||||||
- `file`: filesystem file or object-storage object;
|
|
||||||
- `directory`: filesystem directory or logical prefix;
|
|
||||||
- `symlink`: local filesystem symlink;
|
|
||||||
- `other`: unknown or unsupported native entry type.
|
|
||||||
|
|
||||||
`Stat` returns metadata for one exact logical path. It may report a real filesystem directory, symlink, file, or exact object. It must not synthesize S3-like directory metadata solely because objects exist below a prefix; callers that need prefix existence or destination emptiness must use `HasAny` or `Walk`.
|
|
||||||
|
|
||||||
`Walk` traverses entries below a prefix and calls a callback for each entry. `WalkOptions` should include:
|
|
||||||
|
|
||||||
- whether traversal is recursive;
|
|
||||||
- an optional entry limit for callers that only need to know whether content exists.
|
|
||||||
|
|
||||||
If a callback returns `ErrStopWalk`, traversal stops successfully and `Walk` returns nil. Any other callback error stops traversal and is returned with storage context where practical. If `WalkOptions.Limit` is greater than zero, reaching the limit stops traversal successfully.
|
|
||||||
|
|
||||||
Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal.
|
|
||||||
|
|
||||||
Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them.
|
|
||||||
|
|
||||||
`HasAny` reports whether at least one entry exists below a prefix. It should stop as soon as content is found.
|
|
||||||
|
|
||||||
Source validation must reject symlink entries reported by local `Stat` or `Walk`.
|
|
||||||
|
|
||||||
## Read Behavior
|
|
||||||
|
|
||||||
`ReadFile` reads the whole object into memory and is appropriate for MVP manifest, state, and ordinary artifact handling.
|
|
||||||
|
|
||||||
`OpenReader` returns a stream for callers that need to copy or hash content without requiring a second storage-specific API. Callers must close the returned reader.
|
|
||||||
|
|
||||||
Both read methods must:
|
|
||||||
|
|
||||||
- validate logical paths before backend access;
|
|
||||||
- reject directories, prefixes, symlinks, and unsupported entries;
|
|
||||||
- return typed not-found and invalid-path errors where applicable.
|
|
||||||
|
|
||||||
## Write Behavior
|
|
||||||
|
|
||||||
`WriteOptions` should include:
|
|
||||||
|
|
||||||
- content type, when the destination backend can use it;
|
|
||||||
- overwrite permission;
|
|
||||||
- atomic or staged write preference;
|
|
||||||
- optional known size for stream writes.
|
|
||||||
|
|
||||||
Backends own staging and atomic behavior where practical:
|
|
||||||
|
|
||||||
- Local backend writes to a temporary file in the destination directory and renames or promotes into place.
|
|
||||||
- SSH/SFTP backend should use a temporary remote file and rename where available.
|
|
||||||
- S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata.
|
|
||||||
|
|
||||||
Remote adapters may buffer or spool `WriteFrom` input when needed to satisfy backend requirements such as content length, multipart upload, or retry behavior. Callers that know the stream size should set `SizeKnown` and `Size`.
|
|
||||||
|
|
||||||
If overwrite is false and the target exists, writes should fail with an already-exists error.
|
|
||||||
|
|
||||||
`WriteFile` and `WriteFrom` should return the written `Entry`, including final path and size where available.
|
|
||||||
|
|
||||||
`DeleteOptions` should include:
|
|
||||||
|
|
||||||
- whether missing managed output paths are ignored;
|
|
||||||
- whether empty parent directories may be pruned for filesystem-like backends.
|
|
||||||
|
|
||||||
## Managed Deletion
|
|
||||||
|
|
||||||
The storage interface should expose a guarded managed deletion operation rather than raw recursive delete.
|
|
||||||
|
|
||||||
`DeleteManagedBundle(ctx, bundlePath, managedOutputPaths, opts)` may delete only:
|
|
||||||
|
|
||||||
- files or objects listed in valid `.distributor.json.outputs`;
|
|
||||||
- `.distributor.json` at the destination bundle path;
|
|
||||||
- empty directories created by those files, for filesystem-like backends.
|
|
||||||
|
|
||||||
`managedOutputPaths` are relative to the destination bundle path. The backend validates each path and resolves it under `bundlePath`.
|
|
||||||
|
|
||||||
If `bundlePath == ""`, deletion may remove explicit managed files at the destination root, but must never delete the root itself.
|
|
||||||
|
|
||||||
Prefix or recursive deletion is out of MVP scope. A future force-overwrite stage may add broader behavior, but it must remain explicit and separately documented.
|
|
||||||
|
|
||||||
## Destination Emptiness
|
|
||||||
|
|
||||||
Destination emptiness should use `HasAny(prefix)` and typed not-found behavior. Callers that only need emptiness must not materialize a full recursive traversal.
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
|
|
||||||
- A local destination bundle path is empty when the directory does not exist or exists with no entries.
|
|
||||||
- An S3-compatible prefix is empty when no objects exist below that exact destination bundle prefix.
|
|
||||||
- Entries outside the exact destination bundle path or prefix do not affect emptiness.
|
|
||||||
|
|
||||||
## Error Model
|
|
||||||
|
|
||||||
Storage should expose typed error categories with wrapping context. Callers should use helper predicates rather than string matching.
|
|
||||||
|
|
||||||
Required categories:
|
|
||||||
|
|
||||||
- not found;
|
|
||||||
- already exists;
|
|
||||||
- not empty;
|
|
||||||
- invalid path;
|
|
||||||
- conflict;
|
|
||||||
- permission;
|
|
||||||
- temporary;
|
|
||||||
- unsupported;
|
|
||||||
- unknown.
|
|
||||||
|
|
||||||
Adapters should translate backend-native errors into these categories while preserving useful operation, backend, path, and cause context.
|
|
||||||
|
|
||||||
## Adapter Expectations
|
|
||||||
|
|
||||||
### Local
|
|
||||||
|
|
||||||
The local backend should:
|
|
||||||
|
|
||||||
- constrain all operations beneath the configured root;
|
|
||||||
- reject traversal and absolute logical paths;
|
|
||||||
- report symlinks through metadata;
|
|
||||||
- reject symlink reads for source artifacts;
|
|
||||||
- use staged writes where practical;
|
|
||||||
- perform managed deletion only for explicit managed files and `.distributor.json`;
|
|
||||||
- clean up empty directories created by managed outputs where safe.
|
|
||||||
|
|
||||||
### Fake
|
|
||||||
|
|
||||||
The fake backend should:
|
|
||||||
|
|
||||||
- be in-memory and deterministic;
|
|
||||||
- implement the same logical path validation rules;
|
|
||||||
- support `Stat`, `Walk`, `HasAny`, byte reads and writes, stream reads and writes, managed deletion, and destination emptiness helper behavior;
|
|
||||||
- support configured symlink entries for validation tests;
|
|
||||||
- be used only by tests.
|
|
||||||
|
|
||||||
### SSH/SFTP
|
|
||||||
|
|
||||||
The SSH/SFTP backend should:
|
|
||||||
|
|
||||||
- use native SFTP operations;
|
|
||||||
- enforce the same logical path rules as local storage;
|
|
||||||
- use temporary file plus rename for staged writes where available;
|
|
||||||
- translate remote errors into storage error categories;
|
|
||||||
- avoid exposing SSH or SFTP dependency types through `internal/storage`.
|
|
||||||
|
|
||||||
### S3-Compatible
|
|
||||||
|
|
||||||
The S3-compatible backend should:
|
|
||||||
|
|
||||||
- treat prefixes as object trees, not real directories;
|
|
||||||
- normalize configured prefix plus logical path into object keys;
|
|
||||||
- use object PUT as publish-on-success;
|
|
||||||
- set content type from `WriteOptions`;
|
|
||||||
- implement traversal and emptiness by exact prefix;
|
|
||||||
- use backend pagination for traversal where available;
|
|
||||||
- allow `HasAny` to stop after the first matching object;
|
|
||||||
- constrain managed deletion to listed output objects and `.distributor.json`.
|
|
||||||
|
|
||||||
## Tests and Fixtures
|
|
||||||
|
|
||||||
Storage implementation stages should test:
|
|
||||||
|
|
||||||
- path validation rejects absolute paths, traversal, empty file paths, backslashes, and dot segments;
|
|
||||||
- `Walk` visits backend-rooted logical paths under a prefix and supports recursive traversal;
|
|
||||||
- a materializing helper sorts walk results lexically for deterministic tests and CLI output;
|
|
||||||
- `HasAny` returns quickly for non-empty prefixes without requiring full traversal;
|
|
||||||
- `ReadFile` and `OpenReader` return equivalent bytes;
|
|
||||||
- `WriteFile` and `WriteFrom` honor overwrite and content-type options;
|
|
||||||
- local staged writes do not leave final files on failure where testable;
|
|
||||||
- managed deletion deletes only state-listed files and `.distributor.json`;
|
|
||||||
- managed deletion never deletes destination root or unlisted files;
|
|
||||||
- destination emptiness helper handles missing, empty, and non-empty local paths;
|
|
||||||
- S3-compatible traversal can use pagination without loading a whole prefix into memory;
|
|
||||||
- symlink entries are reported and rejected by source validation;
|
|
||||||
- typed errors are usable through helper predicates;
|
|
||||||
- fake backend behavior matches local backend semantics relevant to core tests.
|
|
||||||
Reference in New Issue
Block a user