Refresh distributor documentation

This commit is contained in:
2026-06-07 23:50:55 +00:00
parent e2529cfdf2
commit 138bc4e7e4
7 changed files with 106 additions and 627 deletions

View File

@@ -1,336 +1,34 @@
# Distributor Integration Roadmap
# Distributor Roadmap
This roadmap describes planned work for adding `distributor` notification
support to `weatherreporter`. The feature is not implemented yet, so this plan
lives under `docs/roadmap/`; non-roadmap documentation should not describe
`notify.distributor` as available until implementation is complete.
Current distributor notification behavior is documented outside the roadmap:
## Purpose
- [Configuration reference](../config.md)
- [Operations guide](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [Distributor adapter internals](../internal/distributor-adapter.md)
`distributor` collects outputs from producer applications and publishes them
through configured downstream pipelines. For `weatherreporter`, the planned
integration should upload generated Markdown weather reports after successful
report generation, without requiring `distributor` to scan or understand the
managed `workspace/` layout.
This file tracks future distributor-related work only.
## Current Repository Facts
## Deferred Enhancements
- `weatherreporter` does not currently implement a notify hook.
- Generated Markdown reports are written to the managed report path returned as
`ReportResult.ReportPath`.
- Optional `--out` and `--out-dir` copies are user-requested extra copies and
are not canonical integration inputs.
- The managed workspace also contains briefing snapshots, metadata, prompt data
packages, and preflight artifacts; these should not be treated as a
distributor source tree.
- The provided distributor docs recommend `pkg/upload.UploadFiles` for
producers that already have generated files, so the initial integration can
upload selected report files directly.
- Add a supported warning-only notification policy.
- Include selected non-report artifacts in uploaded bundles.
- Poll distributor run status after upload acceptance.
- Persist upload retry state across process restarts.
- Add explicit CLI controls for distributor behavior.
## Decisions Locked
## Non-Goals Without A Separate Design
- Add generic configurable secrets-directory support before the distributor
adapter work. This is project infrastructure, not a distributor-only token
helper.
- When a secrets directory is configured, each regular file directly under that
directory maps filename to environment variable name and file contents to
environment variable value.
- Secret-file values overwrite pre-existing environment variables.
- Secrets loading must never log, print, persist, or include secret values in
errors, metadata, batch output, examples, or docs.
- Add a narrow `internal/adapters/distributor` adapter for the external
integration.
- Keep `gitea.maximumdirect.net/eric/distributor/pkg/upload` and
`gitea.maximumdirect.net/eric/distributor/pkg/bundle` types inside that
adapter.
- Upload one distributor bundle per successfully generated report.
- Include only the generated Markdown report in the initial bundle.
- Use the report definition's batch output filename as the default
bundle-relative Markdown path, such as `daily.md`, `tomorrow.md`,
`three-day.md`, `weekend.md`, or `storm.md`.
- Derive the default bundle ID and idempotency key from producer name, location
ID, report ID, and RunID.
- Treat distributor upload failure as a report failure when notification is
enabled; single-report and batch commands should exit nonzero for enabled
upload failures.
- Keep destination routing, Markdown-to-HTML transformation, public URLs, and
nginx directory layout in distributor configuration, not in
`weatherreporter`.
- Do not make distributor scan the weatherreporter workspace.
- Do not move destination routing into weatherreporter.
- Do not move Markdown-to-HTML transformation into weatherreporter.
- Do not store raw bearer tokens in configuration files.
## Planned Configuration
## Required Constraints For Future Work
Add future generic secrets configuration:
```yaml
secrets:
directory: ""
```
Add future distributor configuration under `notify.distributor`:
```yaml
notify:
distributor:
enabled: false
endpoint: https://distributor.example.com
token_env: DISTRIBUTOR_UPLOAD_TOKEN
timeout: 30s
failure_policy: error
bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
idempotency_key_template: "{bundle_id}"
report_path_template: "{batch_output_name}"
```
Planned behavior:
- `secrets.directory` defaults to an empty string, which disables secrets
loading.
- When `secrets.directory` is configured, load only regular files directly
under that directory.
- Each secret filename must be a valid environment variable name matching
`[A-Za-z_][A-Za-z0-9_]*`.
- Reject missing configured directory, invalid filenames, subdirectories,
symlinks, unreadable files, and empty filenames.
- Strip one trailing `\n` or `\r\n` from secret file contents for compatibility
with common mounted-secret formats. Preserve all other bytes.
- `enabled` defaults to `false`.
- `endpoint` is required only when distributor notification is enabled.
- `token_env` is required only when enabled. Raw bearer tokens should not be
stored in config files. The token value is read from normal environment state
after secrets-directory loading has been applied.
- `timeout` controls the upload call timeout and should default to `30s`.
- `failure_policy` should initially support `error`. Reserve `warn` as a
future option only if it is implemented and tested.
- Template variables should be explicit and validated:
`location_id`, `report_id`, `run_id`, `artifact_group`, and
`batch_output_name`.
- `idempotency_key_template` may reference `{bundle_id}` after the bundle ID is
rendered.
- `report_path_template` should be user-visible from the first implementation
because bundle-relative paths can affect downstream distributor routing.
- Rendered template values must produce valid distributor bundle paths: relative
slash-separated paths with no empty path, absolute path, backslash, `.`, `..`,
empty segment, `manifest.json`, or `.distributor.json`.
## Recommended Implementation Stages
### Stage 1: Secrets Directory Support
Goal: add generic file-backed environment secret loading before distributor
notification depends on environment tokens.
Implementation guidance:
- Add a generic `secrets.directory` config field under `internal/config`.
- Default the directory to an empty string so secrets loading is disabled unless
explicitly configured.
- Load secrets after config file parsing and before validation that needs
environment-backed values.
- Map each regular file directly under the configured directory to an
environment variable: filename becomes the variable name and contents become
the value.
- Use secret-file values to overwrite pre-existing environment variables.
- Strip one trailing `\n` or `\r\n`; preserve all other bytes.
- Reject missing directories, invalid environment-variable filenames,
subdirectories, symlinks, unreadable files, and empty filenames.
- Keep secret values out of errors, logs, CLI output, metadata, and examples.
Acceptance criteria:
- Empty `secrets.directory` performs no environment changes.
- Configured secrets directory values are available through normal environment
lookup after config loading.
- Secret files overwrite existing environment variables with the same name.
- Invalid directory entries fail with path/name context but without secret
values.
- Existing config loading behavior remains unchanged when secrets are disabled.
Suggested tests:
- Config defaults leave secrets loading disabled.
- A configured missing secrets directory fails.
- Regular files set environment variables and overwrite existing values.
- Invalid filenames, subdirectories, symlinks, and unreadable files fail.
- One trailing newline or CRLF is stripped; other content is preserved.
- No secret values appear in returned errors.
### Stage 2: Config And Validation
Goal: add disabled-by-default distributor notification configuration.
Implementation guidance:
- Add notify/distributor config structs under `internal/config`.
- Add defaults for disabled notification, timeout, failure policy, bundle ID
template, idempotency key template, and report path template.
- Validate enabled config: endpoint must be an absolute URL, `token_env` must be
non-empty, timeout must be positive, failure policy must be supported, and
templates must use only known variables.
- Keep `token_env` as the configured token selector. Do not add raw token config.
- Update maintained examples and config documentation only after code support is
implemented.
Acceptance criteria:
- Disabled distributor notification requires no endpoint or token env.
- Enabled distributor notification rejects invalid endpoint, missing token env,
nonpositive timeout, unsupported failure policy, unknown template variables,
and invalid rendered report bundle paths.
- Existing config loading behavior and precedence remain unchanged.
Suggested tests:
- Config defaults include disabled distributor notification.
- Example configs load successfully.
- Enabled config validation covers required fields and invalid template cases.
- A distributor token supplied through `secrets.directory` is visible through
the configured `token_env` name after config loading.
### Stage 3: Distributor Adapter
Goal: isolate distributor package usage behind an internal adapter.
Implementation guidance:
- Add `internal/adapters/distributor` with package-owned request and result
types.
- Read the bearer token from the configured environment variable after
secrets-directory loading has populated environment state.
- Use distributor `UploadFiles` with one file:
the managed Markdown report as the source path and the rendered
`report_path_template` as the bundle path.
- Return upload result information that app orchestration can record or expose
without leaking distributor dependency types.
- Wrap upload errors with endpoint, bundle ID, and report path context while
avoiding token exposure.
Acceptance criteria:
- No distributor dependency types leak outside `internal/adapters/distributor`.
- Missing source report path, missing token, invalid bundle path, and upload
failure return actionable errors.
- Idempotency conflict errors are preserved or wrapped clearly enough for
troubleshooting.
Suggested tests:
- Adapter request construction uses the configured endpoint, token, bundle ID,
idempotency key, source report path, and bundle-relative path.
- Token values do not appear in returned errors.
- Upload conflict and generic upload failure produce useful wrapped errors.
- Tokens supplied through the secrets directory are accepted via `token_env`.
### Stage 4: App Notify Hook
Goal: upload successful generated reports through the configured notifier.
Implementation guidance:
- Add an app-owned notifier interface so app tests can use fakes.
- Construct distributor notification requests from the successful
`ReportResult`, resolved report definition, configured location, and
notification config.
- Invoke notification only after Scriptorium report generation succeeds and
final metadata has been saved.
- Do not notify after Weather API, briefing, prompt input, render preflight, or
Scriptorium run failure.
- In batch mode, notify each successful report independently. If notification
fails and distributor notification is enabled, mark that report failed and
make the aggregate batch result nonzero.
Acceptance criteria:
- Disabled notification is a no-op.
- Single-report generation returns an error when enabled upload fails.
- Batch generation continues independent reports, records upload failures per
report, and exits nonzero when any enabled notification fails.
- Existing report artifact paths, optional output copies, and Scriptorium
behavior remain unchanged.
Suggested tests:
- No notifier call occurs before successful report generation.
- Successful notifier call receives the managed report path, not `--out` or
`--out-dir` copy paths.
- Enabled notifier failure affects single-report and batch command outcomes.
- Batch still continues later reports after one notification failure.
### Stage 5: CLI And Output Behavior
Goal: preserve CLI syntax while surfacing notification outcome where useful.
Implementation guidance:
- Do not add distributor-specific CLI flags in the first implementation; use
configuration only.
- Preserve existing command names, flags, help text shape, workspace paths, and
generated Markdown outputs.
- Extend batch JSON report items only if the implementation records useful
notification status, accepted distributor run ID, or notification error.
- Keep stderr status lines concise and avoid exposing secrets.
Acceptance criteria:
- `go run ./cmd/weatherreporter --help` remains accurate.
- Existing generate and run command syntax remains stable.
- Batch summaries clearly indicate notification-caused report failures if
notification status is added.
Suggested tests:
- Existing CLI parser tests continue to pass.
- Batch JSON includes notification fields only when implemented and documented.
- Secret-like token values never appear in CLI output.
### Stage 6: Documentation
Goal: document implemented distributor behavior after the feature exists.
Implementation guidance:
- Update `docs/config.md` with `secrets.directory`,
`notify.distributor` fields, and defaults.
- Update `docs/operations.md` with notification timing, failure behavior, and
the fact that managed report paths are uploaded.
- Update `docs/troubleshooting.md` with invalid secrets directory, missing
token, upload conflict, upload rejection, and distributor unavailable cases.
- Update `docs/internal/app-orchestration.md` to include notify ordering.
- Add `docs/internal/distributor-adapter.md` for the adapter contract.
- Keep distributor API details in `docs/integrations/distributor/`; link there
instead of duplicating the full upstream contract.
Acceptance criteria:
- Non-roadmap docs describe only implemented notification behavior.
- Config examples include no raw tokens.
- Secret handling documentation describes mechanisms, not secret values.
- Documentation clearly distinguishes distributor destination routing from
weatherreporter upload responsibilities.
### Stage 7: Final Validation
Goal: verify the feature without requiring live distributor service access.
Validation commands:
```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
```
Additional validation:
- Run focused config, adapter, app, and CLI tests.
- Confirm maintained example configs load.
- Search for accidental raw-token config examples.
- Confirm `go.mod` includes distributor only after implementation requires it.
- Confirm no test failures leak secret values in error output.
## Open Questions
No questions block this roadmap. The initial implementation should use the
locked defaults above and expose configurable templates for bundle identity and
bundle-relative report path. Secrets-directory support is generic project
infrastructure and should remain useful for future integrations beyond
distributor.
- Distributor package types stay inside `internal/adapters/distributor`.
- Weatherreporter submits explicit source bundles built from generated files.
- Optional `--out` and `--out-dir` copies remain operator conveniences, not
canonical upload sources.
- Secret values stay out of errors, logs, CLI output, metadata, examples, and
documentation.

View File

@@ -49,6 +49,26 @@ These ideas are not current behavior:
Each item needs its own design note before implementation. Non-roadmap docs
must not describe these as available behavior.
## Deferred: Distributor Notification Enhancements
Distributor notification currently uploads one managed Markdown report per
successful generated report through the configured HTTP upload endpoint.
These enhancements are not current behavior:
- `failure_policy: warn`;
- uploading metadata, briefing snapshots, data packages, or preflight artifacts;
- polling distributor status after upload acceptance;
- durable upload retry queues;
- distributor-specific CLI flags;
- making distributor scan the weatherreporter workspace;
- handling destination routing, Markdown-to-HTML transformation, public URLs,
or nginx layout inside weatherreporter.
Any distributor enhancement should preserve the existing adapter boundary:
weatherreporter selects explicit generated files and submits source bundles,
while distributor owns destination routing and publication behavior.
## Deferred: Cleanup Refactors
The initial cleanup pass intentionally left these refactors out because the

View File

@@ -1,297 +1,36 @@
# Distributor Feature Implementation Roadmap
# Distributor Follow-Up Roadmap
This roadmap turns `docs/roadmap/distributor.md` into staged implementation
work for a future coding agent. It is a planning document only. The feature is
not implemented until the stages below are completed and the implemented docs
are updated.
Current distributor notification behavior is documented in:
## Purpose
- [Configuration reference](../config.md)
- [Operations guide](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [App orchestration internals](../internal/app-orchestration.md)
- [Distributor adapter internals](../internal/distributor-adapter.md)
- [Distributor integration contracts](../integrations/distributor/api.md)
Implement generic file-backed environment secrets and distributor notification
support while preserving current CLI syntax, managed artifact paths, RunIDs,
and report output behavior. Distributor package types must stay behind a narrow
adapter boundary, and distributor should receive explicitly selected generated
Markdown files rather than scanning `workspace/`.
This roadmap tracks only future work that is not implemented.
## Locked Decisions
## Deferred Enhancements
- Implement generic `secrets.directory` support before distributor notification.
- Secret files overwrite pre-existing environment variables.
- Never expose secret values in errors, logs, CLI output, metadata, docs, or
examples.
- Add `notify.distributor` config; do not add raw token config.
- Keep public CLI commands and flags unchanged.
- Upload one distributor bundle per successfully generated report.
- Initial distributor bundles include only the managed Markdown report at
`ReportResult.ReportPath`.
- The default bundle-relative Markdown path uses the report definition's
`BatchOutputName`.
- Default bundle ID and idempotency key derive from producer, location ID,
report ID, and RunID.
- Enabled distributor upload failure is a report failure and should produce a
nonzero command exit.
- Support a non-failing distributor notification policy such as
`failure_policy: warn`.
- Upload additional generated artifacts, such as metadata, briefing snapshots,
data packages, or preflight output.
- Poll distributor status after upload acceptance and expose downstream
publication failures.
- Add durable retry queues for upload failures.
- Add distributor-specific CLI controls if operators need per-run behavior that
configuration cannot cover cleanly.
## Stage 1: Secrets Directory Support
## Boundaries To Preserve
Goal: implement generic file-backed environment secret loading in
`internal/config`.
Implementation guidance:
- Add `secrets.directory` config with a built-in default of `""`, meaning
disabled.
- Load secrets after config file parsing and CLI overrides, before validation
completes.
- Read only regular files directly under the configured directory.
- Map each file basename to an environment variable name and file contents to
the value.
- Validate names with `[A-Za-z_][A-Za-z0-9_]*`.
- Reject missing configured directory, invalid filenames, subdirectories,
symlinks, unreadable files, and empty filenames.
- Strip one trailing `\n` or `\r\n`; preserve all other bytes.
- Keep error messages actionable but do not include secret values.
Acceptance criteria:
- Empty `secrets.directory` performs no environment changes.
- Valid secret files set environment variables after config loading.
- Secret file values overwrite existing environment variables.
- Invalid entries fail before application workflows run.
- Existing config loading behavior remains unchanged when secrets are disabled.
Tests:
- Defaults leave secrets loading disabled.
- Valid file sets an environment variable.
- File value overwrites an existing environment variable.
- Missing directory, invalid filename, subdirectory, symlink, and unreadable
file cases fail.
- Newline and CRLF trimming works exactly once.
- Secret values are absent from error strings.
## Stage 2: Notify Configuration And Templates
Goal: add disabled-by-default `notify.distributor` config and template
validation.
Planned config:
```yaml
notify:
distributor:
enabled: false
endpoint: https://distributor.example.com
token_env: DISTRIBUTOR_UPLOAD_TOKEN
timeout: 30s
failure_policy: error
bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
idempotency_key_template: "{bundle_id}"
report_path_template: "{batch_output_name}"
```
Implementation guidance:
- Add notify/distributor config structs, defaults, YAML loading, and validation
under `internal/config`.
- When disabled, omit endpoint and token requirements.
- When enabled, validate absolute endpoint URL, non-empty `token_env`, positive
timeout, `failure_policy=error`, known template variables, and valid rendered
report bundle paths.
- Supported template variables are `location_id`, `report_id`, `run_id`,
`artifact_group`, and `batch_output_name`; `idempotency_key_template` may
also reference `{bundle_id}`.
- Do not add distributor-specific CLI flags.
Acceptance criteria:
- Defaults include disabled distributor notification.
- Enabled config rejects invalid endpoint, token env, timeout, failure policy,
templates, and rendered bundle paths.
- Rendered bundle paths reject absolute paths, backslashes, `.`, `..`, empty
segments, `manifest.json`, and `.distributor.json`.
- Tokens supplied by `secrets.directory` are visible through `token_env` after
config loading.
Tests:
- Config defaults and examples load.
- Disabled distributor config accepts omitted distributor fields.
- Enabled validation covers each invalid field.
- Template rendering and validation are covered with table tests.
## Stage 3: Distributor Adapter
Goal: isolate distributor upload behavior in `internal/adapters/distributor`.
Implementation guidance:
- Add weatherreporter-owned request and result types.
- Keep `gitea.maximumdirect.net/eric/distributor/pkg/upload` and
`gitea.maximumdirect.net/eric/distributor/pkg/bundle` types inside the
adapter.
- Read the bearer token from `token_env` after config/secrets loading.
- Use `UploadFiles` with exactly one file: source path is the managed Markdown
report path and bundle path is the rendered `report_path_template`.
- Return the accepted distributor run ID when available.
- Wrap endpoint, bundle ID, idempotency key, source path, and bundle path
context into errors without exposing token values.
- Preserve idempotency conflict diagnosis.
Acceptance criteria:
- No distributor dependency types leak outside the adapter.
- Missing token, source path, bundle path, or upload client inputs fail with
actionable errors.
- Upload success returns a weatherreporter-owned result.
- Upload failure errors do not include token values.
Tests:
- Fake upload client receives expected endpoint, token, bundle ID, idempotency
key, source path, and bundle path.
- Missing token/source path/bundle path errors are actionable.
- Token values are not present in errors.
- Upload conflict and generic failure are wrapped clearly.
## Stage 4: App Notification Hook
Goal: add app-level notification orchestration after successful report
generation.
Implementation guidance:
- Add an app-owned notifier interface for tests.
- Default to no-op notification when distributor notification is disabled.
- Notify only after successful `scriptorium run` and final metadata save.
- Use `ReportResult.ReportPath` as the distributor source file.
- Never use `--out` or `--out-dir` copies as distributor source files.
- Do not notify after Weather API, briefing, prompt input, render preflight, or
Scriptorium run failure.
- Single-report generation should return an error when enabled notification
fails.
- Batch generation should continue independent reports; notification failure
marks that report failed and makes the aggregate batch nonzero.
Acceptance criteria:
- Disabled notification leaves current generation behavior unchanged.
- Successful notification is recorded in the app result when useful.
- Enabled notification failure fails a single-report command.
- Batch results distinguish generation failure from notification failure enough
for operators to diagnose the failed report.
- Existing artifact paths, output copies, and Scriptorium behavior remain
unchanged.
Tests:
- No notifier call occurs on pre-generation failures.
- Notifier receives the managed report path.
- Enabled notifier success and failure are covered.
- Batch continues after one notification failure and reports aggregate failure.
## Stage 5: CLI Output And Batch Reporting
Goal: keep public CLI syntax stable while surfacing notification outcome if the
app records it.
Implementation guidance:
- Do not add distributor-specific CLI flags.
- Preserve current command names, help shape, generated paths, `--out`, and
`--out-dir`.
- If batch JSON is extended, use explicit fields such as
`notificationStatus`, `notificationRunId`, and `notificationError`.
- Keep stderr status lines concise and secret-free.
Acceptance criteria:
- `weatherreporter --help` remains accurate.
- Existing CLI parser behavior remains stable.
- Batch JSON includes notification details only when implemented and
documented.
- CLI stdout and stderr do not include token values.
Tests:
- Existing CLI parser tests pass, except intentional JSON additions.
- Batch output covers notification success and failure.
- Secret-like token values are absent from CLI output.
## Stage 6: Documentation
Goal: update implemented documentation after the feature exists.
Implementation guidance:
- Update `docs/config.md` with `secrets.directory`, `notify.distributor`,
defaults, and secret handling.
- Update `docs/operations.md` with notification timing, failure behavior, and
managed report upload source.
- Update `docs/troubleshooting.md` with invalid secrets directory, missing
token, upload conflict, upload rejection, and distributor unavailable cases.
- Update `docs/internal/app-orchestration.md` to include notify ordering.
- Add `docs/internal/distributor-adapter.md` for adapter inputs, outputs,
boundaries, config fields, failure behavior, tests, and invariants.
- Keep full upstream distributor contract details in
`docs/integrations/distributor/`.
Acceptance criteria:
- Non-roadmap docs describe only implemented behavior.
- Config examples load and contain no raw secrets.
- Secret handling docs describe mechanisms, not secret values.
- Docs distinguish weatherreporter upload responsibility from distributor
destination routing and Markdown-to-HTML transformation.
Tests and checks:
- Run config example loading tests.
- Search docs and examples for accidental raw tokens.
- Run `git diff --check`.
## Stage 7: Final Validation
Goal: verify behavior without a live distributor service.
Validation commands:
```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
go test ./internal/config ./internal/app ./internal/cli ./internal/adapters/distributor
```
Manual validation:
- No token values appear in errors, CLI output, docs, examples, metadata, or
batch JSON.
- `go.mod` includes distributor only after adapter implementation requires it.
- Existing report paths, RunIDs, and `--out` / `--out-dir` behavior are
unchanged.
- Distributor upload tests use fakes or local test doubles, not a live service.
## Deferred Work
These are out of scope for the initial implementation:
- `failure_policy: warn`.
- Uploading metadata, briefing, data package, or preflight artifacts.
- Distributor status polling after `202 Accepted`.
- Durable upload retry queues.
- Distributor-specific CLI flags.
- Making distributor scan `workspace/`.
- Destination routing, Markdown-to-HTML transformation, public URLs, or nginx
layout in `weatherreporter`.
## Global Validation Checklist
- Preserve current public CLI syntax.
- Preserve managed workspace artifact paths.
- Keep public CLI syntax stable unless a separate CLI design changes it.
- Keep managed workspace artifact paths and RunIDs stable.
- Keep distributor dependency types behind `internal/adapters/distributor`.
- Keep secrets out of user-facing and persisted output.
- Keep examples valid and secret-free.
- Update implemented docs in the same change as implementation.
- Run `go test ./...`, `go run ./cmd/weatherreporter --help`, and
`git diff --check` before finishing implementation.
- Keep raw token values out of errors, CLI output, metadata, docs, examples,
and persisted artifacts.
- Upload explicitly selected generated files; do not make distributor scan the
weatherreporter workspace.
- Leave destination routing, Markdown-to-HTML transformation, public URLs, and
nginx layout to distributor.