Clean up completed documentation roadmaps
This commit is contained in:
@@ -1,593 +0,0 @@
|
|||||||
# Code Quality and Deduplication Audit
|
|
||||||
|
|
||||||
## 1. Executive summary
|
|
||||||
|
|
||||||
Overall code quality is strong. The repository has clear package boundaries, good current-behavior documentation, focused adapter packages, and tests close to most implemented behavior. The most important cleanup opportunities are narrow and behavior-preserving rather than architectural.
|
|
||||||
|
|
||||||
Top three refactoring targets:
|
|
||||||
|
|
||||||
1. HTTP upload admission, body staging, and archive validation are split across `internal/app` and `internal/ingest` in a way that duplicates size and content-type policy and buffers uploads in memory.
|
|
||||||
2. Runtime config loading, default config path selection, secret loading, and warning projection are repeated across app entrypoints.
|
|
||||||
3. Run orchestration mixes destination processing, failure aggregation, warning recording, and report event ordering in one large loop, making future changes harder to review safely.
|
|
||||||
|
|
||||||
The codebase appears ready for a limited cleanup pass. I do not see a major architectural risk that requires a redesign before the next release.
|
|
||||||
|
|
||||||
## 2. Repository map reviewed
|
|
||||||
|
|
||||||
Reviewed policy and current-behavior documentation:
|
|
||||||
|
|
||||||
- `AGENTS.md`
|
|
||||||
- `README.md`
|
|
||||||
- `docs/policy/architecture.md`
|
|
||||||
- `docs/policy/development.md`
|
|
||||||
- `docs/policy/documentation.md`
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- `docs/internal/*.md`
|
|
||||||
- `docs/roadmap/http.md`
|
|
||||||
- `docs/roadmap/implementation.md`
|
|
||||||
|
|
||||||
Reviewed implementation areas:
|
|
||||||
|
|
||||||
- `cmd/distributor`: executable entrypoint.
|
|
||||||
- `internal/cli`: root command, `version`, `run`, `serve`, `validate`, `inspect`, and `manifest create` parsing.
|
|
||||||
- `internal/app`: run orchestration, configured source diagnostics, backend factory, manifest creation, CLI output, HTTP upload server, upload coordinator, and pipeline coordinator.
|
|
||||||
- `internal/config`: config structs, defaults, validation, quantity parsing, S3/SSH helpers, and secrets resolver.
|
|
||||||
- `internal/bundle`: storage-backed source discovery and validation.
|
|
||||||
- `pkg/bundle`: public manifest model, digest logic, manifest building, local validation, and local bundle writer.
|
|
||||||
- `internal/storage` and `internal/storage/fake`: backend interface, path helpers, walk helpers, typed errors, and fake backend.
|
|
||||||
- `internal/adapters/local`, `internal/adapters/ssh`, and `internal/adapters/s3`: runtime storage adapters.
|
|
||||||
- `internal/ingest`: HTTP upload archive staging.
|
|
||||||
- `internal/publish`: destination planning, output selection, link projection, state writing, cleanup, and force replacement.
|
|
||||||
- `internal/state`: destination state parsing, validation, comparison, and JSON projection.
|
|
||||||
- `internal/transform` and `internal/transform/markdown`: transform registry and Markdown rendering.
|
|
||||||
- `internal/link`, `internal/notify`, `internal/logging`, and `internal/testutil`.
|
|
||||||
- `examples`, package tests, and package `testdata`.
|
|
||||||
|
|
||||||
Requested areas that are absent as separate packages:
|
|
||||||
|
|
||||||
- `internal/stage`
|
|
||||||
- `internal/modules`
|
|
||||||
- `internal/validators`
|
|
||||||
- `internal/artifacts`
|
|
||||||
- `internal/manifest`
|
|
||||||
- `internal/schema`
|
|
||||||
- `internal/report`
|
|
||||||
- public `pkg` packages other than `pkg/bundle`
|
|
||||||
|
|
||||||
Those absences are consistent with current architecture policy; the corresponding behavior lives in narrower existing packages.
|
|
||||||
|
|
||||||
## 3. High-confidence deduplication opportunities
|
|
||||||
|
|
||||||
### HTTP upload body handling should be owned by ingestion
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/upload_http.go`
|
|
||||||
- `internal/app/upload_coordinator.go`
|
|
||||||
- `internal/ingest/archive.go`
|
|
||||||
- `internal/app/upload_http_test.go`
|
|
||||||
- `internal/app/upload_http_integration_test.go`
|
|
||||||
- `internal/ingest/archive_test.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- `internal/app/upload_http.go` validates upload content types in `supportedUploadContentType`, while `internal/ingest/archive.go` validates the same content types in `archiveFormat`.
|
|
||||||
- `internal/app/upload_http.go` enforces upload size in `readUploadBody`, while `internal/ingest/archive.go` enforces upload size again in `writeLimited`.
|
|
||||||
- The HTTP handler reads the full upload body into memory before submission, then the coordinator passes a `bytes.Reader` to ingestion.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- The app transport layer now partially owns archive policy that should belong to `internal/ingest`.
|
|
||||||
- Large accepted uploads are buffered in memory even though ingestion already has streaming-to-disk mechanics.
|
|
||||||
- Future archive formats, content types, or upload limit changes would need coordinated edits in multiple packages.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Move supported content-type checking behind an ingestion-owned helper, for example `ingest.ValidateContentType` or `ingest.IsSupportedContentType`.
|
|
||||||
- Change upload admission so the request body is streamed to staging exactly once before the HTTP handler returns `202 Accepted`.
|
|
||||||
- Keep queue-full rejection before reading the body.
|
|
||||||
- Queue a staged local bundle root, not an unread request body. This preserves async distribution while keeping HTTP request lifetime separate from later pipeline execution.
|
|
||||||
- Keep `UploadCoordinator` responsible for queueing, status, per-pipeline serialization, and execution. Keep `internal/ingest` responsible for archive format, size, extraction, cleanup, and source bundle validation.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- HTTP handler rejects full queues without reading the body.
|
|
||||||
- HTTP handler streams a valid body to ingestion and returns `202` only after staging succeeds.
|
|
||||||
- Unsupported content types are rejected through the ingestion-owned content-type policy.
|
|
||||||
- Oversized uploads are rejected without retaining a staged run.
|
|
||||||
- Accepted upload status still transitions through queued/running/succeeded or failed without depending on an open HTTP request body.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Medium. The behavior change is internal but touches admission timing and async execution boundaries. It should be implemented in a focused prompt with existing HTTP integration tests extended first.
|
|
||||||
|
|
||||||
### Runtime config and secret setup should have one app-level helper
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/run.go`
|
|
||||||
- `internal/app/source_select.go`
|
|
||||||
- `internal/app/serve.go`
|
|
||||||
- `internal/app/backends.go`
|
|
||||||
- `internal/app/run_warnings.go`
|
|
||||||
- `internal/config`
|
|
||||||
- `internal/app/*_test.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- Defaulting an empty config path to `config.DefaultConfigPath` appears in `Run`, `RunPipeline`, `RunPipelineWithLocalSource`, and `Serve`.
|
|
||||||
- Config loading and secret loading are separate repeated steps in `buildRunReport`, `selectSourceBundlesFromConfig`, and `Serve`.
|
|
||||||
- Secret conflict warnings are projected in run and configured source diagnostics, while serve loads secrets without using or exposing conflict warning metadata.
|
|
||||||
- Backend factory construction from a config environment is repeated through provider plumbing.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- Config and secret precedence is a public operational policy.
|
|
||||||
- A future change to config discovery, secret conflict reporting, or runtime environment construction could drift between `run`, `serve`, `validate`, and `inspect`.
|
|
||||||
- Tests for secrets and credential resolution need to cover several entrypoints today.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Add a small app-level runtime setup helper, for example `loadRuntimeConfig(optionsConfigPath string) (runtimeConfig, error)`.
|
|
||||||
- The helper should own default config path selection, `config.LoadFile`, `config.LoadSecretEnvironment`, and conversion of secret conflicts into `OutputWarning` values.
|
|
||||||
- Keep config parsing and validation in `internal/config`; the helper should not duplicate config policy.
|
|
||||||
- Let `run`, configured `validate`/`inspect`, and `serve` call the helper and then apply command-specific behavior.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- One focused app test proving default config path selection remains unchanged where injection permits it.
|
|
||||||
- Existing secret conflict JSON/text warning tests for `run`, `validate`, and `inspect`.
|
|
||||||
- Serve startup test proving duplicate and missing upload tokens still fail without leaking values.
|
|
||||||
- S3 explicit credential tests proving the resolver is still used through the helper.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Low to medium. This is a straightforward centralization but touches several command entrypoints.
|
|
||||||
|
|
||||||
### Run destination processing needs a narrow helper boundary
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/run.go`
|
|
||||||
- `internal/app/run_output.go`
|
|
||||||
- `internal/app/run_failures.go`
|
|
||||||
- `internal/app/run_selection.go`
|
|
||||||
- `internal/publish`
|
|
||||||
- `internal/app/run_test.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- Destination backend open failures and publish planning/execution failures each manually add `runFailures`, record summary failure counts, append `RunActionRecord`, and append pipeline event indexes.
|
|
||||||
- `publish.Build` error handling patches missing `Plan` identity fields inline before converting the plan to a run action.
|
|
||||||
- Fixed-path warning emission is interleaved with destination selection and publish plan handling.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- `run --format json` depends on exact action ordering, warnings, partial failures, and summary counters.
|
|
||||||
- Future changes to actions, links, notifications, or HTTP upload reports could accidentally update one failure path but not another.
|
|
||||||
- The current loop is correct but dense enough that small behavior changes are hard to review.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Extract a narrow `runDestination` or `destinationRunner` helper that processes one destination and returns action records, warnings, summary deltas, and failures.
|
|
||||||
- Add a helper for recording a destination-scoped failure that updates `runFailures`, `runSummary`, `RunReport.Actions`, and pipeline events in one place.
|
|
||||||
- Add a helper that normalizes partial `publish.Plan` identity fields before action projection.
|
|
||||||
- Do not introduce a generic workflow engine or stage abstraction.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Preserve existing run text output golden assertions.
|
|
||||||
- Preserve JSON partial-result behavior when planning fails after destination processing begins.
|
|
||||||
- Add one focused test where destination open fails for multiple selected bundles and verify action records, output errors, and summary counters stay aligned.
|
|
||||||
- Add one fixed-path dry-run warning test after extraction to verify event ordering.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Medium. The refactor is behavior-preserving but touches the most important user-facing workflow.
|
|
||||||
|
|
||||||
### Archive path validation duplicates source path policy with a different error surface
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/ingest/archive.go`
|
|
||||||
- `pkg/bundle/path.go`
|
|
||||||
- `internal/storage/path.go`
|
|
||||||
- `internal/ingest/archive_test.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- `cleanArchivePath`, `pkg/bundle.ValidateSourcePath`, and `storage.ValidatePath` all enforce clean slash-separated relative paths with no backslashes, no absolute paths, and no dot segments.
|
|
||||||
- Archive staging needs slightly different policy because directories are allowed and `manifest.json` is allowed only at the root, so the duplication is not completely mechanical.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- Path safety is high-risk behavior.
|
|
||||||
- Future changes to source path rules could miss archive extraction, especially around backslashes, reserved names, or dot segments.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Keep archive-specific rules in `internal/ingest`, but use a shared path-checking primitive where possible.
|
|
||||||
- A good shape is an exported `pkg/bundle.ValidatePathSegmented` only if it fits the public producer API, or an internal helper in ingestion that delegates file-entry validation to `pkg/bundle.ValidateSourcePath` for regular files after handling directory-specific exceptions.
|
|
||||||
- Preserve current archive-specific errors and tests.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Table tests shared or mirrored across bundle path validation and archive path cleaning for absolute paths, traversal, backslashes, dot segments, empty names, root `manifest.json`, nested `manifest.json`, and `.distributor.json`.
|
|
||||||
- Regression tests proving directories are still accepted in archives but symlinks and hardlinks remain rejected.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Low to medium. Path validation changes need careful tests, but the desired change can be small.
|
|
||||||
|
|
||||||
## 4. Medium-confidence opportunities
|
|
||||||
|
|
||||||
### Source and destination backend config shapes could expose a normalized view
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/config/config.go`
|
|
||||||
- `internal/config/defaults.go`
|
|
||||||
- `internal/config/validate.go`
|
|
||||||
- `internal/app/backends.go`
|
|
||||||
- `internal/config/*_test.go`
|
|
||||||
- `internal/app/backends_test.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- `config.Backend` and `config.Destination` duplicate backend fields for local, SSH, S3, and credentials.
|
|
||||||
- Defaults for source backends and destination backends are implemented in separate functions.
|
|
||||||
- App backend opening converts both shapes into `backendOpenSpec`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- New backend fields must be added to both YAML structs, defaulting paths, validation paths, app open-spec conversion, docs, and tests.
|
|
||||||
- The current pattern is easy to understand but likely to drift as more backend-specific fields are added.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Keep the YAML shape unchanged for compatibility.
|
|
||||||
- Add package-local helpers in `internal/config` that return a normalized backend view for either source or destination.
|
|
||||||
- Use that view for shared backend defaulting and validation where it improves clarity.
|
|
||||||
- Keep destination-only fields such as `publish`, `transfer`, `links`, and `path_mapping` on `Destination`.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing source and destination backend validation tests should continue to pass.
|
|
||||||
- Add a table test that validates equivalent local, SSH, and S3 source/destination backend field requirements through the shared view.
|
|
||||||
- Add a test that `http_upload` remains source-only.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Medium. This reduces future drift, but the current duplication is understandable and does not need to be the first cleanup.
|
|
||||||
|
|
||||||
### CLI command scaffolding is mostly shared, but manifest create has special parsing
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/cli/run.go`
|
|
||||||
- `internal/cli/serve.go`
|
|
||||||
- `internal/cli/source_mode.go`
|
|
||||||
- `internal/cli/manifest.go`
|
|
||||||
- `internal/cli/version.go`
|
|
||||||
- `internal/cli/root_test.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- Several commands repeat `flag.NewFlagSet`, `SetOutput`, help handling, format parsing, and usage exit handling.
|
|
||||||
- `manifest create` uses `splitManifestCreateArgs` to allow a positional bundle path before flags, unlike Go's default `flag` behavior.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- CLI syntax and error behavior are public.
|
|
||||||
- A broad CLI helper could accidentally obscure command-specific parsing, but a narrow helper could reduce repeated setup and invalid-format handling.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Do not introduce a CLI framework.
|
|
||||||
- Consider a tiny helper for common `FlagSet` creation and output-format parsing after higher-value app/config cleanup.
|
|
||||||
- Keep `manifest create` custom parsing local unless another command needs the same interspersed positional behavior.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Preserve current CLI usage-error tests.
|
|
||||||
- Add explicit tests for `manifest create <path> --id x`, `manifest create --id x <path>`, and invalid missing flag values before any parser cleanup.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Low if kept narrow; medium if over-generalized.
|
|
||||||
|
|
||||||
### Output DTOs repeat bundle metadata projection
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/validate.go`
|
|
||||||
- `internal/app/inspect.go`
|
|
||||||
- `internal/app/manifest.go`
|
|
||||||
- `internal/app/run_output.go`
|
|
||||||
- `internal/app/output.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- `inspect` and `manifest create` both project bundle file metadata into command-specific JSON structs.
|
|
||||||
- `validate`, `inspect`, and `manifest create` each define local result types and file record types.
|
|
||||||
- RFC3339 formatting uses both `time.RFC3339` and the equivalent literal layout string.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- JSON output is now a public interface.
|
|
||||||
- Repeated projection can drift in field names, timestamp formatting, or path display rules.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Add a small app-local projection helper for bundle summaries and manifest file records.
|
|
||||||
- Use `time.RFC3339` instead of literal RFC3339 layouts.
|
|
||||||
- Keep command-specific result structs where the command output semantics differ.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- JSON structural tests for `validate`, `inspect`, and `manifest create` before and after the helper.
|
|
||||||
- A timestamp-format assertion using an offset timestamp to confirm current behavior is preserved.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Low.
|
|
||||||
|
|
||||||
### Duplicate-run coordination overlaps conceptually with upload coordination
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/upload_coordinator.go`
|
|
||||||
- `docs/internal/app.md`
|
|
||||||
- `internal/app/upload_coordinator_test.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- Both coordinators define run records, statuses, timestamps, status transitions, context handling, and active pipeline protection.
|
|
||||||
- The upload coordinator additionally queues, stages, expires status records, and serializes same-pipeline upload execution.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
- The concepts are similar enough to confuse future contributors.
|
|
||||||
- However, the behavior is not identical: one rejects duplicate active runs, while the other queues accepted uploads.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
- Do not merge the coordinators now.
|
|
||||||
- Review whether duplicate-run coordination is still needed as an exported
|
|
||||||
app-level helper. If it is intended for future transports, document that role
|
|
||||||
clearly. If not, remove it and its tests in a separate dead-code cleanup.
|
|
||||||
- If both remain, extract only tiny shared timestamp/status helpers if a real third coordinator appears.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- If retained, keep existing duplicate-run tests.
|
|
||||||
- If removed, run `go test ./internal/app ./internal/cli` and verify no current behavior depended on it.
|
|
||||||
|
|
||||||
Risk level:
|
|
||||||
|
|
||||||
- Low for documentation clarification, medium for removal because it is exported from an internal package and documented for maintainers.
|
|
||||||
|
|
||||||
## 5. Boundary and responsibility concerns
|
|
||||||
|
|
||||||
The major boundaries are sound:
|
|
||||||
|
|
||||||
- CLI parsing stays in `internal/cli`.
|
|
||||||
- Config defaults and validation stay in `internal/config`.
|
|
||||||
- Backend-specific filesystem, SFTP, and S3 behavior stays in adapters.
|
|
||||||
- Manifest semantics are centralized in `pkg/bundle`, with `internal/bundle` adding storage-backed discovery and validation.
|
|
||||||
- Destination state comparison stays in `internal/state`.
|
|
||||||
- Publish planning/execution stays in `internal/publish`.
|
|
||||||
- Transform implementation is behind `internal/transform`.
|
|
||||||
|
|
||||||
Concerns worth addressing:
|
|
||||||
|
|
||||||
- HTTP upload request-body staging currently crosses the app/ingest boundary. The app transport layer should not own body buffering and archive size enforcement beyond admission and HTTP status projection.
|
|
||||||
- Runtime config setup is app-layer behavior, but it is repeated rather than named. A runtime setup helper would clarify the boundary between `internal/config` and command-specific execution.
|
|
||||||
- `internal/app/run.go` owns too many destination-loop details. Extracting a destination processing helper would keep orchestration in app while reducing local complexity.
|
|
||||||
|
|
||||||
Recommended homes:
|
|
||||||
|
|
||||||
- Upload archive policy: `internal/ingest`.
|
|
||||||
- HTTP route/auth/status mapping: `internal/app/upload_http.go`.
|
|
||||||
- Queueing/status/execution: `internal/app/upload_coordinator.go`.
|
|
||||||
- Runtime config plus secret setup: a small helper in `internal/app`, using `internal/config`.
|
|
||||||
- Path and state filenames: keep in `internal/storage`.
|
|
||||||
|
|
||||||
## 6. Path, key, and naming construction review
|
|
||||||
|
|
||||||
Centralized and healthy areas:
|
|
||||||
|
|
||||||
- `storage.StateFileName`, `storage.StatePath`, `storage.ManagedBundleTargets`, `storage.Join`, `storage.DisplayPath`, and logical path validation are used in core publication and tests.
|
|
||||||
- S3 object-key mapping is contained in `internal/adapters/s3`.
|
|
||||||
- SSH and local native path conversion stay inside their adapters.
|
|
||||||
- Link URL construction is isolated in `internal/publish/links.go` and URL validation in `internal/link`.
|
|
||||||
- Manifest name and schema version are centralized in `pkg/bundle`, with `internal/bundle` aliases.
|
|
||||||
|
|
||||||
Areas needing cleanup:
|
|
||||||
|
|
||||||
- Archive path cleaning duplicates much of source/storage path policy and should either delegate to a shared primitive or be tightly covered by mirrored tests.
|
|
||||||
- Upload run ID construction is isolated, but the shape is partly policy. Keep tests around `<pipeline_id>.<utc_timestamp>.<random_suffix>` before changing coordinator code.
|
|
||||||
- Some app tests still construct destination state and source paths locally. `internal/testutil` already covers many cases; additional helper use should be opportunistic, not a sweeping test rewrite.
|
|
||||||
|
|
||||||
## 7. Resolution and catalog review
|
|
||||||
|
|
||||||
Named concept resolution is mostly consistent:
|
|
||||||
|
|
||||||
- Backend names are defined in `internal/config/defaults.go`.
|
|
||||||
- Runtime backend construction is app-owned through `backendFactory` and the storage registry.
|
|
||||||
- Transform names are defined in `internal/transform`, and app wiring owns concrete registration.
|
|
||||||
- Publish/transform policy combinations use `config.ValidatePublishTransformPolicy`.
|
|
||||||
- Configured source selection for `validate` and `inspect` is shared in `source_select.go`.
|
|
||||||
|
|
||||||
Potential refinements:
|
|
||||||
|
|
||||||
- A normalized backend config view would make backend field resolution less repetitive across source and destination config.
|
|
||||||
- Transform and backend registries should remain separate; there is no evidence that a generic registry abstraction would help.
|
|
||||||
- No separate catalog package is needed for the current feature set.
|
|
||||||
|
|
||||||
## 8. Config and command-loading review
|
|
||||||
|
|
||||||
Config loading is reliable and strict:
|
|
||||||
|
|
||||||
- YAML known-field checking is enabled.
|
|
||||||
- Defaults are applied before validation.
|
|
||||||
- Validation collects multiple field errors.
|
|
||||||
- Secrets are loaded without mutating `os.Environ`.
|
|
||||||
- Explicit S3 credential references use the config-owned resolver.
|
|
||||||
|
|
||||||
Likely accidental duplication:
|
|
||||||
|
|
||||||
- Default config path selection and `config.LoadFile` are repeated in several app entrypoints.
|
|
||||||
- Secret loading is repeated in run, source diagnostics, and serve.
|
|
||||||
- Secret conflict warning projection is not represented by one runtime setup result.
|
|
||||||
|
|
||||||
Intentional differences:
|
|
||||||
|
|
||||||
- `serve` loads upload tokens and does not produce CLI JSON output.
|
|
||||||
- `validate` and `inspect` support local-path shortcut mode, while `run` and `serve` are config-driven.
|
|
||||||
- `manifest create` is local filesystem producer tooling and does not load app config.
|
|
||||||
|
|
||||||
Recommended cleanup:
|
|
||||||
|
|
||||||
- Centralize runtime config and secret setup in `internal/app`.
|
|
||||||
- Keep CLI flag parsing local to command files.
|
|
||||||
- Keep `manifest create` outside runtime config loading.
|
|
||||||
|
|
||||||
## 9. State, manifest, or progress handling review
|
|
||||||
|
|
||||||
Manifest handling is in good shape:
|
|
||||||
|
|
||||||
- `pkg/bundle` owns manifest parsing, digest grammar, source path validation, canonical bundle digest, local manifest building, and local bundle writing.
|
|
||||||
- `internal/bundle` delegates normalized manifest semantics to `pkg/bundle` and adds storage-backed validation.
|
|
||||||
- Destination state embeds the normalized manifest and validates through `internal/bundle`/`pkg/bundle`.
|
|
||||||
|
|
||||||
State handling is in good shape:
|
|
||||||
|
|
||||||
- `.distributor.json` parsing, validation, JSON projection, and comparison live in `internal/state`.
|
|
||||||
- Publish execution writes destination state only after outputs are written.
|
|
||||||
- Managed replacement deletes only state-listed outputs plus `.distributor.json`; forced replacement is explicit and bounded.
|
|
||||||
|
|
||||||
Progress/status handling:
|
|
||||||
|
|
||||||
- `RunReport` is the core run result model and supports JSON partial-result output.
|
|
||||||
- HTTP upload status is memory-only and documented as such.
|
|
||||||
- Duplicate-run coordination and upload coordination overlap conceptually but
|
|
||||||
have different policies. Avoid merging unless product behavior converges.
|
|
||||||
|
|
||||||
Gaps:
|
|
||||||
|
|
||||||
- HTTP upload staging currently stores the request body in memory before queueing. This is both a quality gap and a mismatch with the intended ingestion boundary.
|
|
||||||
- There is no durable upload status, but this is documented as deferred work and should not be addressed in cleanup.
|
|
||||||
|
|
||||||
## 10. Refactors to avoid
|
|
||||||
|
|
||||||
Avoid these changes in the cleanup pass:
|
|
||||||
|
|
||||||
- Do not introduce a generic workflow engine or stage framework. The current explicit workflow is easier to audit.
|
|
||||||
- Do not add a CLI framework. The standard-library CLI is sufficient and policy-approved.
|
|
||||||
- Do not merge local, SSH, S3, and fake adapters behind a shared implementation layer. Their semantics differ enough that generic helpers would likely hide important behavior.
|
|
||||||
- Do not collapse `pkg/bundle` and `internal/bundle`. The public producer API and storage-backed distributor validation have different responsibilities.
|
|
||||||
- Do not move destination state comparison into `publish` or app orchestration.
|
|
||||||
- Do not redesign JSON output envelopes while doing cleanup.
|
|
||||||
- Do not add durable queues, retry workers, HTTP TLS, zstd, or browser UI under the banner of refactoring. These are feature work.
|
|
||||||
- Do not rewrite tests wholesale to use a new fixture system. Add helpers only where they reduce immediate duplication around changed code.
|
|
||||||
|
|
||||||
## 11. Recommended implementation sequence
|
|
||||||
|
|
||||||
1. HTTP upload staging boundary cleanup.
|
|
||||||
- Move supported content-type policy to `internal/ingest`.
|
|
||||||
- Stop buffering accepted uploads in `upload_http.go`.
|
|
||||||
- Queue staged bundle roots rather than request bodies.
|
|
||||||
- Extend HTTP upload tests first.
|
|
||||||
|
|
||||||
2. Runtime config setup helper.
|
|
||||||
- Add an app-level helper for default config path, config load, secret load, environment resolver, and secret warnings.
|
|
||||||
- Use it from `run`, configured `validate`/`inspect`, and `serve` where applicable.
|
|
||||||
- Preserve command-specific behavior.
|
|
||||||
|
|
||||||
3. Run destination processing extraction.
|
|
||||||
- Add small helpers for destination-scoped failure recording and plan identity normalization.
|
|
||||||
- Extract one-destination processing only if the helper remains readable.
|
|
||||||
- Preserve action ordering and report output.
|
|
||||||
|
|
||||||
4. Backend config normalized view.
|
|
||||||
- Add source/destination backend view helpers in `internal/config`.
|
|
||||||
- Use them for defaulting and validation if tests show the shape remains clear.
|
|
||||||
- Keep YAML structs and public config unchanged.
|
|
||||||
|
|
||||||
5. Bundle output projection cleanup.
|
|
||||||
- Add app-local helpers for file record and bundle summary projection.
|
|
||||||
- Use `time.RFC3339` consistently.
|
|
||||||
- Preserve command-specific JSON field names.
|
|
||||||
|
|
||||||
6. Archive/source path validation test alignment.
|
|
||||||
- Add mirrored path safety tests around ingestion and bundle path validation.
|
|
||||||
- Only centralize code if the helper does not blur archive directory semantics.
|
|
||||||
|
|
||||||
7. Coordinator intent cleanup.
|
|
||||||
- Decide whether duplicate-run coordination is retained for internal future use.
|
|
||||||
- If retained, clarify comments/docs. If removed, do it as a separate dead-code commit.
|
|
||||||
|
|
||||||
8. Test helper cleanup.
|
|
||||||
- Expand `internal/testutil` only for repeated setup touched by the previous refactors.
|
|
||||||
- Avoid moving every test fixture.
|
|
||||||
|
|
||||||
## 12. Test strategy
|
|
||||||
|
|
||||||
Tests to add before refactoring:
|
|
||||||
|
|
||||||
- HTTP upload handler test proving queue-full rejection does not consume the body.
|
|
||||||
- HTTP upload test proving accepted upload staging completes before `202 Accepted`.
|
|
||||||
- Ingestion content-type policy tests exposed through the new helper.
|
|
||||||
- Run report test covering destination open failure for multiple selected bundles.
|
|
||||||
- CLI JSON tests for `inspect` and `manifest create` timestamp formatting before projection cleanup.
|
|
||||||
|
|
||||||
Tests to run with each cleanup stage:
|
|
||||||
|
|
||||||
- HTTP upload cleanup: `go test ./internal/ingest ./internal/app ./internal/cli`
|
|
||||||
- Config setup cleanup: `go test ./internal/config ./internal/app ./internal/cli`
|
|
||||||
- Run processing cleanup: `go test ./internal/app ./internal/publish ./internal/state`
|
|
||||||
- Backend config view cleanup: `go test ./internal/config ./internal/app`
|
|
||||||
- Output projection cleanup: `go test ./internal/app ./internal/cli`
|
|
||||||
- Path validation cleanup: `go test ./pkg/bundle ./internal/bundle ./internal/ingest ./internal/storage`
|
|
||||||
- Final cleanup validation: `go test ./...`
|
|
||||||
|
|
||||||
Useful read-only checks:
|
|
||||||
|
|
||||||
- `rg -n "LoadFile\\(|LoadSecretEnvironment\\(|DefaultConfigPath" internal/app internal/cli`
|
|
||||||
- `rg -n "application/x-tar|application/gzip|application/x-gzip" internal docs`
|
|
||||||
- `rg -n "2006-01-02T15:04:05Z07:00" internal pkg`
|
|
||||||
- `rg -n "manifest.json|\\.distributor.json|StatePath|DisplayPath" internal pkg`
|
|
||||||
|
|
||||||
## 13. Appendix: findings not worth acting on
|
|
||||||
|
|
||||||
Adapter `ReadFile` and `WriteFile` wrappers:
|
|
||||||
|
|
||||||
- Local, SSH, S3, and fake backends each implement byte helpers in terms of stream helpers. This is small duplication but appropriate because each adapter owns error translation and metadata semantics.
|
|
||||||
|
|
||||||
Adapter traversal implementation:
|
|
||||||
|
|
||||||
- Local filesystem walking, SFTP walking, and S3 pagination look similar at the interface level but are semantically different. Keep traversal mechanics in adapters and shared callback behavior in `storage.WalkEmitter`.
|
|
||||||
|
|
||||||
State and manifest raw JSON parsing:
|
|
||||||
|
|
||||||
- `pkg/bundle` and `internal/state` both parse raw JSON with pointer fields to detect missing required fields. The schemas and error contexts differ, so a generic required-field parser would not be worth the complexity.
|
|
||||||
|
|
||||||
CLI help text:
|
|
||||||
|
|
||||||
- Help text repeats command names and flags. This is acceptable in a small hand-written CLI and keeps command files readable.
|
|
||||||
|
|
||||||
Test fixture strings:
|
|
||||||
|
|
||||||
- Some tests inline YAML snippets or expected output strings despite `internal/testutil`. Inline data is often clearer for edge cases. Only centralize fixture setup when tests are already being changed for a behavior-preserving refactor.
|
|
||||||
|
|
||||||
HTTP JSON response helpers:
|
|
||||||
|
|
||||||
- HTTP API responses use simple JSON objects rather than the CLI JSON envelope. This is intentional because HTTP status codes and route-specific responses are not the same public interface as CLI command output.
|
|
||||||
|
|
||||||
Public and internal bundle validation:
|
|
||||||
|
|
||||||
- `pkg/bundle.ValidateBundle` is local-filesystem producer validation; `internal/bundle.Validate` is storage-backed distributor validation. Keep both, with shared manifest semantics delegated through `pkg/bundle`.
|
|
||||||
@@ -1,399 +0,0 @@
|
|||||||
# Code Quality Cleanup Roadmap
|
|
||||||
|
|
||||||
## Current Baseline
|
|
||||||
|
|
||||||
The codebase has completed the local, SSH/SFTP, S3, public bundle package,
|
|
||||||
manifest creation, JSON output, path mapping, link generation, and HTTP upload
|
|
||||||
work documented in the current user and internal docs.
|
|
||||||
|
|
||||||
The audit in `docs/roadmap/audit.md` found no major architectural risk. The
|
|
||||||
remaining cleanup work should be narrow, behavior-preserving, and focused on
|
|
||||||
reducing drift in upload staging, runtime config setup, run reporting, backend
|
|
||||||
config handling, output projection, path validation tests, and internal
|
|
||||||
coordination code.
|
|
||||||
|
|
||||||
One intentional behavior change is part of this cleanup roadmap: malformed
|
|
||||||
authenticated upload archives should be rejected before `202 Accepted`, rather
|
|
||||||
than accepted and later marked failed. Valid staged uploads should still run
|
|
||||||
asynchronously after admission.
|
|
||||||
|
|
||||||
## Cleanup Principles
|
|
||||||
|
|
||||||
- Preserve public CLI behavior, config schema, manifest schema, destination
|
|
||||||
state schema, backend behavior, and JSON envelopes unless a stage explicitly
|
|
||||||
says otherwise.
|
|
||||||
- Keep config parsing and validation in `internal/config`.
|
|
||||||
- Keep CLI parsing in `internal/cli`.
|
|
||||||
- Keep upload archive policy in `internal/ingest`; keep HTTP routing,
|
|
||||||
authentication, and status projection in `internal/app`.
|
|
||||||
- Keep backend-specific filesystem, SSH/SFTP, and S3 behavior in adapter
|
|
||||||
packages.
|
|
||||||
- Prefer small package-local helpers over broad abstractions.
|
|
||||||
- Add or strengthen tests before refactoring behavior that affects public
|
|
||||||
output, upload admission, path safety, or run reporting.
|
|
||||||
|
|
||||||
## Active Cleanup Stages
|
|
||||||
|
|
||||||
Implement these stages in order. Each stage should be small enough for one
|
|
||||||
implementation prompt and should leave the repository passing the listed focused
|
|
||||||
tests before moving to the next stage.
|
|
||||||
|
|
||||||
## Stage 1: HTTP Upload Staging Boundary
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Move archive validation and upload body staging fully behind `internal/ingest`,
|
|
||||||
stop app-layer full-body buffering, and reject malformed archives before
|
|
||||||
returning `202 Accepted`.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Add an ingestion-owned content-type helper, such as
|
|
||||||
`ValidateContentType(contentType string) error`, and remove duplicated
|
|
||||||
content-type policy from the HTTP handler.
|
|
||||||
- Replace the current handler-side `readUploadBody` buffering with streaming
|
|
||||||
staging through `internal/ingest`.
|
|
||||||
- Introduce a two-step upload coordinator admission model:
|
|
||||||
- reserve a run id and queue slot before consuming the request body;
|
|
||||||
- stage and validate the archive using that reserved run id;
|
|
||||||
- enqueue only a successfully staged local bundle root for async execution.
|
|
||||||
- Keep queue-full rejection before reading the body.
|
|
||||||
- Preserve `401` for missing or invalid bearer tokens, `415` for unsupported
|
|
||||||
content type, `413` for oversized uploads, and `503` for a full queue.
|
|
||||||
- Return a pre-acceptance `400` for malformed tar/gzip content or invalid
|
|
||||||
staged bundles.
|
|
||||||
- Preserve async queued/running/succeeded/failed status after a valid staged
|
|
||||||
bundle is accepted.
|
|
||||||
- Do not add durable queues, idempotency keys, zstd, or new routes.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- Update `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
|
|
||||||
`docs/troubleshooting.md`, `docs/internal/app.md`, and
|
|
||||||
`docs/internal/ingest.md` only as needed to describe the new
|
|
||||||
pre-acceptance failure boundary.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/ingest ./internal/app ./internal/cli`
|
|
||||||
- Queue-full upload rejection does not read the request body.
|
|
||||||
- Unsupported content type is rejected through ingestion-owned policy.
|
|
||||||
- Oversized uploads return `413` and do not retain a staged run.
|
|
||||||
- Malformed tar/gzip content returns `400` before a run id is issued.
|
|
||||||
- Valid tar and tar.gz uploads return `202` after staging and still transition
|
|
||||||
through async status.
|
|
||||||
- HTTP responses and status records do not leak bearer tokens or secret values.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- `internal/app` no longer buffers the full upload body before staging.
|
|
||||||
- A valid accepted upload has a committed staged bundle root before the `202`
|
|
||||||
response is sent.
|
|
||||||
- Invalid archive content cannot create an accepted run id.
|
|
||||||
|
|
||||||
## Stage 2: Runtime Config And Secret Setup Helper
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Centralize runtime config path resolution, config loading, secret loading,
|
|
||||||
environment resolver creation, and secret-conflict warning projection in one
|
|
||||||
app-layer helper.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Add a small `internal/app` runtime setup helper that:
|
|
||||||
- defaults an empty config path to `config.DefaultConfigPath`;
|
|
||||||
- calls `config.LoadFile`;
|
|
||||||
- calls `config.LoadSecretEnvironment`;
|
|
||||||
- exposes the loaded config, config path, `config.Environment`, and
|
|
||||||
`[]OutputWarning` for secret conflicts.
|
|
||||||
- Use the helper from `Run`, `RunPipeline`, `RunPipelineWithLocalSource`,
|
|
||||||
configured `Validate`/`Inspect`, and `Serve` where applicable.
|
|
||||||
- Keep `manifest create` outside runtime config loading.
|
|
||||||
- Keep YAML structs, defaults, validation, and secret-directory parsing in
|
|
||||||
`internal/config`.
|
|
||||||
- Preserve app test injection points for backend factories and upload handler
|
|
||||||
tests.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- Update `docs/internal/app.md` if helper boundaries or flow descriptions
|
|
||||||
change. User-facing docs should not change unless observable behavior changes.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/config ./internal/app ./internal/cli`
|
|
||||||
- Default config path behavior remains unchanged.
|
|
||||||
- Secret conflict warnings still appear in text and JSON output for `run`,
|
|
||||||
configured `validate`, and configured `inspect`.
|
|
||||||
- `serve` still fails startup safely for missing, empty, or duplicate upload
|
|
||||||
tokens without leaking values.
|
|
||||||
- Explicit S3 credential references still resolve through the config-owned
|
|
||||||
environment resolver.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- Runtime commands no longer repeat config path defaulting and secret loading.
|
|
||||||
- Config policy remains owned by `internal/config`.
|
|
||||||
|
|
||||||
## Stage 3: Run Destination Processing Extraction
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Reduce complexity in the main run loop while preserving run report behavior,
|
|
||||||
warning ordering, action ordering, failure aggregation, and text/JSON output.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Extract narrow helpers from `internal/app/run.go` for destination-scoped
|
|
||||||
processing.
|
|
||||||
- Centralize destination-scoped failure recording so one helper updates
|
|
||||||
`runFailures`, `runSummary`, `RunReport.Actions`, and pipeline events.
|
|
||||||
- Centralize normalization of partial `publish.Plan` identity fields before
|
|
||||||
converting plans to run action records.
|
|
||||||
- Keep app orchestration explicit; do not introduce a generic workflow engine,
|
|
||||||
stage framework, or broad runner abstraction.
|
|
||||||
- Preserve independent destination fan-out and partial-result behavior.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- Update `docs/internal/app.md` only if helper names or package layout
|
|
||||||
descriptions materially change.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/app ./internal/publish ./internal/state`
|
|
||||||
- Destination open failures for multiple selected bundles keep action records,
|
|
||||||
output errors, summary counters, and pipeline events aligned.
|
|
||||||
- JSON partial-result output remains unchanged when destination planning or
|
|
||||||
execution fails after a report exists.
|
|
||||||
- Fixed-path dry-run warnings appear in the same order as before.
|
|
||||||
- Existing run text output assertions continue to pass.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- `run.go` delegates destination-scoped record/failure bookkeeping to helpers.
|
|
||||||
- No public output shape or ordering changes.
|
|
||||||
|
|
||||||
## Stage 4: Backend Config Normalized View
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Reduce source/destination backend config drift while preserving the current YAML
|
|
||||||
schema and public config behavior.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Add package-local normalized backend view helpers in `internal/config` for
|
|
||||||
source and destination backend fields.
|
|
||||||
- Use the normalized view to reduce duplication in backend defaulting and
|
|
||||||
validation where it remains clearer than the current paired code.
|
|
||||||
- Keep `config.Backend` and `config.Destination` YAML structs and tags
|
|
||||||
unchanged.
|
|
||||||
- Preserve destination-only policy fields on `Destination`.
|
|
||||||
- Preserve `http_upload` as source-only and invalid for destinations.
|
|
||||||
- Update app backend opening only if the normalized view provides clearer
|
|
||||||
handoff without leaking config internals.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- None expected unless internal docs mention the old paired implementation
|
|
||||||
shape in a way that becomes misleading.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/config ./internal/app`
|
|
||||||
- Equivalent local, SSH, and S3 source/destination validation remains
|
|
||||||
consistent.
|
|
||||||
- Defaults for SSH port/host key policy, S3 region/prefix/force-path-style, and
|
|
||||||
HTTP upload staging fields remain unchanged.
|
|
||||||
- `http_upload` remains valid only for sources.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- Adding a future backend field has one obvious defaulting/validation path.
|
|
||||||
- Public config files and examples continue to load unchanged.
|
|
||||||
|
|
||||||
## Stage 5: Command Output Projection Cleanup
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Reduce drift in bundle and file metadata projection for app command JSON
|
|
||||||
results.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Add small app-local projection helpers for bundle summaries and manifest file
|
|
||||||
records used by `validate`, `inspect`, and `manifest create`.
|
|
||||||
- Use `time.RFC3339` consistently instead of equivalent literal layouts.
|
|
||||||
- Preserve existing JSON envelope fields, command names, command-specific result
|
|
||||||
field names, text output, and fatal error behavior.
|
|
||||||
- Do not redesign CLI JSON output or HTTP JSON responses.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- None expected unless tests reveal current docs are stale.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/app ./internal/cli`
|
|
||||||
- JSON output for `validate`, `inspect`, and `manifest create` remains
|
|
||||||
structurally stable.
|
|
||||||
- RFC3339 timestamps remain unchanged, including offset-preserving source
|
|
||||||
timestamps where current behavior preserves them.
|
|
||||||
- Text output remains unchanged.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- Bundle/file projection logic is shared where semantics match.
|
|
||||||
- Command-specific result structs remain easy to read.
|
|
||||||
|
|
||||||
## Stage 6: Archive And Source Path Validation Alignment
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Protect path safety by aligning archive path tests with source and storage path
|
|
||||||
policy, without blurring archive-specific rules.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Add mirrored path-safety table tests around `internal/ingest`, `pkg/bundle`,
|
|
||||||
`internal/bundle`, and `internal/storage` where useful.
|
|
||||||
- Keep archive-specific directory handling, root-level manifest rules, duplicate
|
|
||||||
file rejection, symlink rejection, hardlink rejection, and special-entry
|
|
||||||
rejection in `internal/ingest`.
|
|
||||||
- Centralize code only if the helper can preserve clear archive semantics and
|
|
||||||
current error behavior.
|
|
||||||
- Do not add new public `pkg/bundle` APIs unless the existing public API cannot
|
|
||||||
safely support the needed shared behavior.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- None expected unless implementation changes error boundaries or internal
|
|
||||||
package descriptions.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./pkg/bundle ./internal/bundle ./internal/ingest ./internal/storage`
|
|
||||||
- Absolute paths, traversal, backslashes, empty paths, dot segments, nested
|
|
||||||
manifests, `.distributor.json` handling, symlinks, hardlinks, devices, and
|
|
||||||
sockets remain covered.
|
|
||||||
- Archive directories remain accepted where safe.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- Path safety policy has regression coverage across archive staging, source
|
|
||||||
bundle validation, and storage logical path validation.
|
|
||||||
- Any code sharing is smaller and clearer than the duplicated logic it replaces.
|
|
||||||
|
|
||||||
## Stage 7: Pipeline Run Coordinator Removal
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Remove the currently unused duplicate-run coordinator to avoid maintaining two
|
|
||||||
similar coordination concepts.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Delete the duplicate-run coordinator, its run record and duplicate-run error
|
|
||||||
types, related helpers, and their tests.
|
|
||||||
- Remove or rewrite `docs/internal/app.md` sections that describe the removed
|
|
||||||
coordinator.
|
|
||||||
- Keep `UploadCoordinator`; do not merge upload queueing with the removed
|
|
||||||
duplicate-run coordinator.
|
|
||||||
- Before deletion, confirm with search that production code does not reference
|
|
||||||
the duplicate-run coordinator constructor, type, or error.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- Update `docs/internal/app.md` because it currently documents the coordinator
|
|
||||||
as an internal implemented component.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/app ./internal/cli`
|
|
||||||
- Search `internal` and `docs` for the removed duplicate-run coordinator symbols;
|
|
||||||
should show no stale references after removal.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- No production, test, or internal documentation references remain for the
|
|
||||||
removed coordinator.
|
|
||||||
- Upload coordination behavior is unchanged.
|
|
||||||
|
|
||||||
## Stage 8: Narrow CLI And Test Helper Cleanup
|
|
||||||
|
|
||||||
Goal:
|
|
||||||
|
|
||||||
Apply only low-risk CLI setup and test fixture cleanup that remains useful after
|
|
||||||
the earlier stages.
|
|
||||||
|
|
||||||
Implementation scope:
|
|
||||||
|
|
||||||
- Add tiny CLI helpers for repeated `flag.FlagSet` setup or output-format
|
|
||||||
parsing only where command behavior remains obvious.
|
|
||||||
- Keep the standard-library CLI; do not introduce a CLI framework.
|
|
||||||
- Keep `manifest create` interspersed positional parsing local unless another
|
|
||||||
command now needs the same parsing behavior.
|
|
||||||
- Expand `internal/testutil` only for repeated setup touched by earlier stages.
|
|
||||||
- Do not rewrite tests wholesale just to use shared helpers.
|
|
||||||
|
|
||||||
Current-behavior documentation updates:
|
|
||||||
|
|
||||||
- None expected unless CLI help or syntax changes. This stage should avoid such
|
|
||||||
changes.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- `go test ./internal/cli ./internal/app`
|
|
||||||
- CLI usage-error tests remain stable.
|
|
||||||
- `manifest create <path> --id x`, `manifest create --id x <path>`, missing
|
|
||||||
flag values, invalid `--format`, and help output remain covered.
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- Remaining CLI/test cleanup is small, readable, and behavior-preserving.
|
|
||||||
- No public CLI syntax or output changes.
|
|
||||||
|
|
||||||
## Refactors To Avoid
|
|
||||||
|
|
||||||
- Do not introduce a generic workflow engine or stage framework.
|
|
||||||
- Do not add a CLI framework.
|
|
||||||
- Do not merge local, SSH, S3, and fake backend adapter implementations.
|
|
||||||
- Do not collapse `pkg/bundle` and `internal/bundle`.
|
|
||||||
- Do not move destination state comparison into `internal/publish` or
|
|
||||||
`internal/app`.
|
|
||||||
- Do not redesign CLI JSON envelopes.
|
|
||||||
- Do not change HTTP JSON response shapes except where Stage 1 requires
|
|
||||||
pre-acceptance error behavior.
|
|
||||||
- Do not add durable upload queues, retry workers, zstd support, in-app TLS,
|
|
||||||
idempotency keys, browser UI, or other feature work.
|
|
||||||
- Do not rewrite tests wholesale to use new fixture helpers.
|
|
||||||
|
|
||||||
## Validation
|
|
||||||
|
|
||||||
After each implementation stage, run the stage-specific tests listed above.
|
|
||||||
|
|
||||||
After all cleanup stages:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./...
|
|
||||||
```
|
|
||||||
|
|
||||||
Recommended consistency checks:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
rg -n "LoadFile\\(|LoadSecretEnvironment\\(|DefaultConfigPath" internal/app internal/cli
|
|
||||||
rg -n "application/x-tar|application/gzip|application/x-gzip" internal docs
|
|
||||||
rg -n "2006-01-02T15:04:05Z07:00" internal pkg
|
|
||||||
rg -n "duplicate-run coordinator" internal docs
|
|
||||||
```
|
|
||||||
|
|
||||||
The cleanup is complete when:
|
|
||||||
|
|
||||||
- all staged tests and `go test ./...` pass;
|
|
||||||
- current-behavior docs describe the implemented Stage 1 upload failure
|
|
||||||
boundary;
|
|
||||||
- `docs/roadmap/audit.md` findings have either been addressed or consciously
|
|
||||||
left in place as noted in this cleanup roadmap;
|
|
||||||
- no completed cleanup behavior is documented only as future work.
|
|
||||||
@@ -2,566 +2,84 @@
|
|||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
This roadmap defines the work required to bring this repository's documentation into compliance with `docs/policy/documentation.md` and the current implementation. It is a planning artifact only: future implementation passes should rewrite or create the target documentation files in staged changes, while keeping unimplemented or aspirational material under `docs/roadmap/`.
|
This roadmap tracks the remaining work required to verify that project documentation complies with `docs/policy/documentation.md` and accurately reflects the current implementation.
|
||||||
|
|
||||||
## Repository Documentation Inventory
|
The documentation migration has rewritten the current user, operator, integration, and internal component docs. This file now records only remaining validation work. Current behavior belongs outside `docs/roadmap/`; deferred or unimplemented work belongs under `docs/roadmap/`.
|
||||||
|
|
||||||
Current documentation and examples found during inspection:
|
## Current Documentation Set
|
||||||
|
|
||||||
- `README.md` — keep and lightly update. It is already short and orientation-focused, but the final pass should verify the quickstart command, links, and wording against the rewritten canonical docs.
|
Current documentation outside roadmap:
|
||||||
- `docs/cli.md` — keep and rewrite. It is the canonical CLI reference, but it mixes command reference, workflows, HTTP upload details, output schema, and recovery links; rewrite around current command behavior in `internal/cli` and output behavior in `internal/app`.
|
|
||||||
- `docs/config.md` — keep and rewrite. It is the canonical user-facing config reference and contains most implemented fields, but it is long and should be rebuilt from `internal/config` structs, defaults, validation, examples, and backend wiring.
|
|
||||||
- `docs/operations.md` — keep and rewrite. It is the canonical operations/recovery doc, but it currently carries broad reference material that should link to `docs/cli.md`, `docs/config.md`, and component docs instead of duplicating them.
|
|
||||||
- `docs/troubleshooting.md` — keep and rewrite. It exists and is symptom-oriented, but it is large enough to require a clean pass against current error behavior and should avoid becoming another operations manual.
|
|
||||||
- `docs/policy/architecture.md` — keep and lightly update only if the code/docs rewrite reveals stale architectural statements. It is a controlling policy document, not user documentation.
|
|
||||||
- `docs/policy/development.md` — keep and lightly update. It is the contributor workflow policy and should remain concise; verify commands and package list after documentation migration.
|
|
||||||
- `docs/policy/documentation.md` — keep unchanged unless documentation policy itself needs a separate policy change. It is controlling for this roadmap.
|
|
||||||
- `docs/internal/app.md` — keep and rewrite. It reflects recent upload coordination and run orchestration, but should be normalized to the required internal-doc sections from `docs/policy/documentation.md`.
|
|
||||||
- `docs/internal/bundle.md` — keep and lightly update. It is close to the implemented component contract; verify path rules, nested manifest handling, and `pkg/bundle` boundary.
|
|
||||||
- `docs/internal/config.md` — keep and lightly update. It is close to current config internals; verify defaults, HTTP upload source-only behavior, secret resolver behavior, and backend support.
|
|
||||||
- `docs/internal/ingest.md` — keep and lightly update. It matches archive staging behavior after cleanup; verify tar entry types, root manifest rules, failure cleanup, and limits.
|
|
||||||
- `docs/internal/link.md` — keep and lightly update. It is concise and maps to `internal/link`.
|
|
||||||
- `docs/internal/notify.md` — keep and lightly update. It should remain explicit that only the no-op notifier is implemented.
|
|
||||||
- `docs/internal/publish.md` — keep and rewrite. It is important and mostly current, but should be checked against `internal/publish` safety, link, transform, and force behavior.
|
|
||||||
- `docs/internal/state.md` — keep and lightly update. It should be checked against state schema, URL validation, comparison outcomes, and output record rules.
|
|
||||||
- `docs/internal/storage.md` — keep and rewrite or split. It covers storage plus local, SSH, S3, and fake backend behavior; consider keeping shared storage in this file and moving external backend protocol notes to integration docs.
|
|
||||||
- `docs/internal/transform.md` — keep and lightly update. It should link to the Markdown integration doc and reflect only implemented transforms.
|
|
||||||
- `docs/integrations/markdown.md` — keep and lightly update. It is the only current integration doc; verify it against `internal/transform/markdown` and `go.mod` Goldmark dependency.
|
|
||||||
- `docs/roadmap/audit.md` — keep or delete after migration review. It is roadmap-scoped and may remain as an audit artifact, but completed cleanup findings should not be the only record of current behavior.
|
|
||||||
- `docs/roadmap/cleanup.md` — delete or replace with a concise status note after all cleanup has landed. It describes completed implementation work and should not remain as the canonical description of current behavior.
|
|
||||||
- `docs/roadmap/http.md` — delete or archive under roadmap only if it still contains deferred HTTP upload work. Implemented HTTP upload behavior belongs in current docs; stale planning prose should not be retained as current guidance.
|
|
||||||
- `docs/roadmap/implementation.md` — delete or archive under roadmap only if it still contains deferred feature work. Implemented behavior belongs in current docs.
|
|
||||||
- `docs/roadmap/documentation.md` — create. This file is the action plan for the documentation migration.
|
|
||||||
- `examples/local-to-local.yml` — keep and lightly update. Minimal local config example; should remain load-tested.
|
|
||||||
- `examples/local-publish.yml` — keep and lightly update. Runnable local source publication example used by README, CLI, and operations quickstarts.
|
|
||||||
- `examples/local-html.yml` — keep and lightly update. Runnable local sidecar HTML publication example.
|
|
||||||
- `examples/local-index.yml` — keep and lightly update. Runnable local `index.html` publication example.
|
|
||||||
- `examples/fan-out.yml` — keep and lightly update. Runnable local fan-out example.
|
|
||||||
- `examples/archive-and-latest.yml` — keep and lightly update. Runnable archive plus fixed latest example.
|
|
||||||
- `examples/http-upload-local.yml` — keep and lightly update. Runnable local HTTP upload server config when a token is supplied.
|
|
||||||
- `examples/ssh-destination.yml` — keep and lightly update. Environment-gated SSH/SFTP example; should remain load-tested but not presented as runnable without editing endpoint details.
|
|
||||||
- `examples/s3-destination.yml` — keep and lightly update. Environment-gated S3-compatible example; should remain load-tested but not presented as runnable without editing endpoint and credentials.
|
|
||||||
- `examples/source-bundle/manifest.json`, `examples/source-bundle/report.md`, `examples/source-bundle/summary.txt` — keep and lightly update only if manifest rules or fixture content changes. This is the maintained source bundle example used by examples and docs.
|
|
||||||
|
|
||||||
No current documentation file obviously needs to be moved before rewriting. New integration docs should be created under `docs/integrations/` where the integration contract is important and already implemented.
|
- `README.md`: concise project orientation and quickstart.
|
||||||
|
- `docs/cli.md`: canonical CLI command, flag, workflow, and output reference.
|
||||||
|
- `docs/config.md`: canonical YAML configuration reference.
|
||||||
|
- `docs/operations.md`: operating, safety, state, upload, and recovery guidance.
|
||||||
|
- `docs/troubleshooting.md`: symptom-oriented diagnostic and safe-fix guide.
|
||||||
|
- `docs/policy/architecture.md`: architecture and invariant policy.
|
||||||
|
- `docs/policy/development.md`: contributor and coding workflow policy.
|
||||||
|
- `docs/policy/documentation.md`: controlling documentation policy.
|
||||||
|
- `docs/integrations/*.md`: implemented external/file-format/protocol contracts.
|
||||||
|
- `docs/internal/*.md`: implemented internal component contracts.
|
||||||
|
- `examples/*.yml` and `examples/source-bundle/*`: maintained example configs and source bundle fixture.
|
||||||
|
|
||||||
## Policy Compliance Assessment
|
Current roadmap files:
|
||||||
|
|
||||||
Required documents for this modular, CLI/config-driven, stateful orchestration project are present:
|
- `docs/roadmap/documentation.md`: this remaining documentation validation plan.
|
||||||
|
- `docs/roadmap/http.md`: deferred HTTP upload extensions only.
|
||||||
|
|
||||||
|
Removed completed roadmap artifacts:
|
||||||
|
|
||||||
|
- `docs/roadmap/audit.md`
|
||||||
|
- `docs/roadmap/cleanup.md`
|
||||||
|
- `docs/roadmap/implementation.md`
|
||||||
|
|
||||||
|
## Remaining Documentation Validation
|
||||||
|
|
||||||
|
Goal: verify the rewritten docs against tests, examples, code, links, and the documentation policy checklist.
|
||||||
|
|
||||||
|
Files to create, update, delete, or move: fixes only if validation finds gaps.
|
||||||
|
|
||||||
|
Repository areas to inspect:
|
||||||
|
|
||||||
- `README.md`
|
- `README.md`
|
||||||
- `docs/policy/architecture.md`
|
|
||||||
- `docs/policy/development.md`
|
|
||||||
- `docs/cli.md`
|
- `docs/cli.md`
|
||||||
- `docs/config.md`
|
- `docs/config.md`
|
||||||
- `docs/operations.md`
|
- `docs/operations.md`
|
||||||
- `docs/internal/`
|
|
||||||
|
|
||||||
Recommended documents are also present:
|
|
||||||
|
|
||||||
- `docs/troubleshooting.md`
|
- `docs/troubleshooting.md`
|
||||||
|
- `docs/internal/`
|
||||||
|
- `docs/integrations/`
|
||||||
|
- `docs/policy/`
|
||||||
- `examples/`
|
- `examples/`
|
||||||
|
- CLI parser code under `internal/cli`
|
||||||
Recommended integration coverage is incomplete. `docs/integrations/markdown.md` exists, but the project also has implemented integration contracts for source bundle manifests, destination state files, SSH/SFTP, S3-compatible storage, and the HTTP upload API. These are documented in current user/internal docs, but not consistently in `docs/integrations/`.
|
- config loading/defaulting/validation under `internal/config`
|
||||||
|
- app/report/upload behavior under `internal/app`
|
||||||
Potential compliance issues to resolve during the rewrite:
|
- source bundle, state, publish, storage, adapter, and transform packages
|
||||||
|
|
||||||
- Several current docs are accurate enough to use as input, but too much reference material is duplicated across `docs/cli.md`, `docs/config.md`, `docs/operations.md`, and `docs/internal/*.md`.
|
Acceptance criteria:
|
||||||
- Completed roadmap docs under `docs/roadmap/` may read as active implementation plans. Keep only roadmap content that is still future/deferred, or explicitly convert completed roadmap material into current docs and delete obsolete plans.
|
|
||||||
- `README.md` is concise, but should link only to canonical current docs and avoid implying that roadmap content is required to understand implemented behavior.
|
- Tests pass for the full repository.
|
||||||
- `docs/config.md` is the right canonical home for config fields, defaults, and examples, but should be rebuilt from `internal/config/config.go`, `internal/config/defaults.go`, and `internal/config/validate.go` so field lists and defaults remain defensible.
|
- Maintained example configs load.
|
||||||
- `docs/cli.md` should be validated against command help strings and parser tests in `internal/cli`, including `manifest create <path> --id x`, `manifest create --id x <path>`, missing flag values, invalid `--format`, and help output.
|
- CLI examples and flags match parser behavior.
|
||||||
- `docs/operations.md` should focus on operating, recovery, safety, storage layout, retries, and HTTP upload lifecycle, not full flag or config reference.
|
- Config fields and defaults match `internal/config`.
|
||||||
- `docs/troubleshooting.md` should remain symptom-first and should link to canonical config/CLI/operations docs rather than repeating broad reference sections.
|
- Operations and troubleshooting docs describe implemented behavior only.
|
||||||
- `docs/internal/storage.md` currently includes adapter details that may be clearer as short integration docs for SSH/SFTP and S3-compatible storage, while keeping internal storage interface boundaries in `docs/internal/storage.md`.
|
- Internal docs preserve package boundaries and policy-required sections.
|
||||||
- Examples are present and load-tested by `internal/config/load_test.go`; future docs should link to those examples and preserve the distinction between runnable local examples and environment-gated remote examples.
|
- Integration docs describe only implemented contracts.
|
||||||
- Link verification is manual today; no automated Markdown link checker was found.
|
- Roadmap files contain only remaining or deferred work.
|
||||||
|
- Links resolve.
|
||||||
No clear non-roadmap current document was found describing obviously unimplemented features as available. However, the rewrite should explicitly check for stale planning terms such as `future`, `planned`, `deferred`, `experimental`, `deprecated`, and completed roadmap-only feature descriptions outside `docs/roadmap/`.
|
- No secrets or private data are present.
|
||||||
|
|
||||||
Roadmap file disposition for the documentation migration:
|
Suggested validation commands:
|
||||||
|
|
||||||
- `docs/roadmap/http.md` should remain for now. It already describes only deferred HTTP upload extensions and links current behavior back to canonical docs.
|
```sh
|
||||||
- `docs/roadmap/implementation.md` should remain for now. It duplicates some deferred HTTP upload extension tracking, so the final roadmap hygiene pass should either merge it into `docs/roadmap/http.md` or delete it after confirming no unique deferred work is lost.
|
go test ./...
|
||||||
- `docs/roadmap/audit.md` should remain until the current docs are rewritten. It is a historical audit artifact, not current behavior, and should be deleted or reduced in the final roadmap hygiene pass if it no longer contains useful deferred work.
|
go test ./internal/config ./internal/cli ./internal/app
|
||||||
- `docs/roadmap/cleanup.md` should remain until the current docs are rewritten. It is a completed cleanup plan and should be deleted in the final roadmap hygiene pass after all implemented cleanup behavior is represented in current docs.
|
go test ./pkg/bundle ./internal/bundle ./internal/state ./internal/publish ./internal/storage ./internal/storage/fake
|
||||||
- `docs/roadmap/documentation.md` should remain as the active documentation migration plan until this documentation migration is complete.
|
go test ./internal/adapters/local ./internal/adapters/ssh ./internal/adapters/s3 ./internal/ingest ./internal/transform/markdown
|
||||||
|
rg -n -i "future|planned|deferred|experimental|deprecated|not implemented|old behavior" README.md docs --glob '!docs/roadmap/**'
|
||||||
## Target Documentation Set
|
rg -n -i "Stage|Phase" README.md docs --glob '!docs/roadmap/**'
|
||||||
|
rg -n -- "--config|--dry-run|--force|--format|--pipeline|--bundle|--id|--file|--created|--overwrite" docs/cli.md internal/cli
|
||||||
The desired final documentation tree should include the files below.
|
rg -n "examples/" README.md docs examples internal/config/load_test.go
|
||||||
|
```
|
||||||
### `README.md`
|
|
||||||
|
|
||||||
- Audience: users, administrators, operators.
|
|
||||||
- Purpose: project orientation and quickest useful command.
|
|
||||||
- Canonical scope: concise project purpose, elevator pitch, shortest useful command, links to canonical docs.
|
|
||||||
- Recommended outline:
|
|
||||||
- `# distributor`
|
|
||||||
- one-paragraph description
|
|
||||||
- shortest useful command using `examples/local-publish.yml`
|
|
||||||
- short list of links to CLI, config, operations, troubleshooting, and producer/integration contract docs
|
|
||||||
- Source-of-truth repository areas: `cmd/distributor/main.go`, `internal/cli`, `examples/local-publish.yml`, `docs/policy/documentation.md`.
|
|
||||||
- Acceptance criteria: under one page; no full config reference; no future behavior outside a short pointer that roadmap content exists under `docs/roadmap/`; all links resolve.
|
|
||||||
|
|
||||||
### `docs/cli.md`
|
|
||||||
|
|
||||||
- Audience: users, administrators, operators.
|
|
||||||
- Purpose: canonical command syntax, flags, workflows, and output behavior.
|
|
||||||
- Canonical scope: CLI command reference and command-level workflows only.
|
|
||||||
- Recommended outline:
|
|
||||||
- shortest useful command
|
|
||||||
- command overview
|
|
||||||
- flag reference by command
|
|
||||||
- mode rules for `validate` and `inspect`
|
|
||||||
- `manifest create` positional parsing examples
|
|
||||||
- common workflows
|
|
||||||
- output and exit behavior
|
|
||||||
- links to config, operations, troubleshooting
|
|
||||||
- Source-of-truth repository areas: `internal/cli/*.go`, `internal/cli/root_test.go`, `internal/app/output.go`, `internal/app/run_output.go`, `internal/app/validate.go`, `internal/app/inspect.go`, `internal/app/manifest.go`.
|
|
||||||
- Acceptance criteria: every documented flag exists in `internal/cli`; no undocumented command appears; examples match parser tests; output descriptions match app tests; no config field reference beyond command usage and links.
|
|
||||||
|
|
||||||
### `docs/config.md`
|
|
||||||
|
|
||||||
- Audience: administrators, operators, advanced users.
|
|
||||||
- Purpose: canonical user-facing YAML configuration reference.
|
|
||||||
- Canonical scope: config file location, minimal config, production-oriented config, field reference, defaults, secrets, maintained examples.
|
|
||||||
- Recommended outline:
|
|
||||||
- config file location and precedence for `--config`
|
|
||||||
- minimal local config
|
|
||||||
- production local config
|
|
||||||
- HTTP upload source config
|
|
||||||
- backend reference: local, SSH, S3, HTTP upload source
|
|
||||||
- publish, transform, path mapping, links, transfer policy
|
|
||||||
- field reference
|
|
||||||
- size and duration values
|
|
||||||
- defaults
|
|
||||||
- secrets and credential resolution
|
|
||||||
- examples list
|
|
||||||
- Source-of-truth repository areas: `internal/config/config.go`, `internal/config/defaults.go`, `internal/config/validate.go`, `internal/config/load.go`, `internal/config/secrets.go`, `internal/app/backends.go`, `internal/app/serve.go`, `internal/config/load_test.go`, examples.
|
|
||||||
- Acceptance criteria: every field listed exists in config structs; every default matches `ApplyDefaults`; unknown-field rejection is documented; `http_upload` is source-only; examples load with `go test ./internal/config`; no execution support is claimed beyond local, SSH/SFTP, S3, and HTTP upload server behavior.
|
|
||||||
|
|
||||||
### `docs/operations.md`
|
|
||||||
|
|
||||||
- Audience: administrators/operators.
|
|
||||||
- Purpose: operating workflows, storage layout, safety, recovery, and upload lifecycle.
|
|
||||||
- Canonical scope: how to operate implemented workflows safely, not exhaustive CLI/config reference.
|
|
||||||
- Recommended outline:
|
|
||||||
- normal validation and dry-run workflow
|
|
||||||
- local publication workflow
|
|
||||||
- HTTP upload workflow and lifecycle
|
|
||||||
- filesystem/storage layout
|
|
||||||
- destination state and retry behavior
|
|
||||||
- dry-run and forced replacement safety
|
|
||||||
- remote backend operational notes
|
|
||||||
- logs/output interpretation
|
|
||||||
- safe cleanup and recovery procedures
|
|
||||||
- links to CLI, config, troubleshooting
|
|
||||||
- Source-of-truth repository areas: `internal/app/run.go`, `internal/app/upload_coordinator.go`, `internal/app/upload_http.go`, `internal/publish`, `internal/state`, `internal/storage`, adapter packages, `internal/app/run_test.go`, `internal/app/upload_*_test.go`.
|
|
||||||
- Acceptance criteria: describes only implemented operational behavior; does not duplicate full config reference; force/delete warnings align with publish and storage tests; upload status retention and memory-only behavior match code.
|
|
||||||
|
|
||||||
### `docs/troubleshooting.md`
|
|
||||||
|
|
||||||
- Audience: administrators/operators.
|
|
||||||
- Purpose: symptom-oriented diagnostics and safe fixes.
|
|
||||||
- Canonical scope: common observed failure modes, causes, diagnostic commands, safe fixes, and links.
|
|
||||||
- Recommended outline:
|
|
||||||
- config load and unknown field failures
|
|
||||||
- source bundle validation failures
|
|
||||||
- destination state/conflict/unmanaged content failures
|
|
||||||
- forced replacement safety failures
|
|
||||||
- SSH host key/auth failures
|
|
||||||
- S3 credential/connectivity/path failures
|
|
||||||
- HTTP upload authentication/content/queue/status failures
|
|
||||||
- JSON output partial-result behavior
|
|
||||||
- Source-of-truth repository areas: error strings and tests in `internal/config`, `pkg/bundle`, `internal/bundle`, `internal/publish`, `internal/state`, `internal/adapters/ssh`, `internal/adapters/s3`, `internal/app/upload_*_test.go`, `internal/cli/root_test.go`.
|
|
||||||
- Acceptance criteria: entries follow symptom/likely cause/diagnostic/safe fix/link format; no broad reference duplication; no secret values; links resolve.
|
|
||||||
|
|
||||||
### `docs/policy/architecture.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: controlling architecture policy and invariants.
|
|
||||||
- Canonical scope: development architecture, boundaries, package ownership, safety invariants.
|
|
||||||
- Recommended outline: keep existing outline unless a separate architecture-policy update is needed.
|
|
||||||
- Source-of-truth repository areas: package layout, current implemented workflow, policy decisions in tests.
|
|
||||||
- Acceptance criteria: remains policy-oriented; does not become user docs; any examples are representative and current.
|
|
||||||
|
|
||||||
### `docs/policy/development.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: contributor workflow and coding conventions.
|
|
||||||
- Canonical scope: repo layout, commands, coding rules, test/doc expectations.
|
|
||||||
- Recommended outline: keep existing outline and verify package list and test commands.
|
|
||||||
- Source-of-truth repository areas: repository tree, `go.mod`, tests, examples, AGENTS.md.
|
|
||||||
- Acceptance criteria: commands work; package list matches current tree; documentation policy links remain accurate.
|
|
||||||
|
|
||||||
### `docs/policy/documentation.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: controlling documentation policy.
|
|
||||||
- Canonical scope: documentation layout, audience, canonical homes, maintenance rules.
|
|
||||||
- Recommended outline: no migration changes unless policy itself changes.
|
|
||||||
- Source-of-truth repository areas: policy intent only.
|
|
||||||
- Acceptance criteria: unchanged in implementation passes unless explicitly requested.
|
|
||||||
|
|
||||||
### `docs/internal/app.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: implemented app orchestration, command use cases, runtime setup, run reports, upload coordination, and boundaries.
|
|
||||||
- Canonical scope: `internal/app` only.
|
|
||||||
- Recommended outline:
|
|
||||||
- purpose
|
|
||||||
- inputs and outputs
|
|
||||||
- boundaries
|
|
||||||
- config fields used
|
|
||||||
- adapters used
|
|
||||||
- run/report behavior
|
|
||||||
- upload server and coordinator behavior
|
|
||||||
- failure behavior
|
|
||||||
- tests to inspect
|
|
||||||
- invariants
|
|
||||||
- Source-of-truth repository areas: `internal/app/*.go`, especially `runtime.go`, `run.go`, `run_destination.go`, `run_output.go`, `serve.go`, `upload_coordinator.go`, `upload_http.go`, app tests.
|
|
||||||
- Acceptance criteria: no references to removed duplicate-run coordinator; upload behavior matches staging-before-acceptance; app docs link to config/CLI docs for user syntax.
|
|
||||||
|
|
||||||
### `docs/internal/bundle.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: source bundle discovery and validation internals.
|
|
||||||
- Canonical scope: `internal/bundle` and its boundary with `pkg/bundle`.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; source manifest behavior; discovery; validation; boundaries; failure behavior; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/bundle`, `pkg/bundle`, `internal/bundle/*_test.go`, `pkg/bundle/*_test.go`.
|
|
||||||
- Acceptance criteria: path safety and reserved manifest/state rules match code; nested manifest rejection is documented; producer API boundary is clear.
|
|
||||||
|
|
||||||
### `docs/internal/config.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: config loading/defaulting/validation internals and secret resolver boundary.
|
|
||||||
- Canonical scope: `internal/config` behavior, not full user config reference.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; loading flow; defaults ownership; validation responsibilities; executable support boundary; secrets; failure behavior; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/config/*.go`, `internal/config/*_test.go`, `internal/app/runtime.go`, `internal/app/backends.go`.
|
|
||||||
- Acceptance criteria: links to `docs/config.md` for user-facing field reference; no duplicate full field table; source-only HTTP upload boundary is clear.
|
|
||||||
|
|
||||||
### `docs/internal/ingest.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: upload archive staging internals.
|
|
||||||
- Canonical scope: `internal/ingest` only.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; accepted content types; extraction rules; bundle validation; cleanup/failure behavior; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/ingest/archive.go`, `internal/ingest/archive_test.go`, `pkg/bundle` validation tests.
|
|
||||||
- Acceptance criteria: documents root manifest rule, nested manifest rejection, duplicate file rejection, symlink/hardlink/special-entry rejection, safe directories, and limits exactly as implemented.
|
|
||||||
|
|
||||||
### `docs/internal/link.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: shared URL validation policy.
|
|
||||||
- Canonical scope: `internal/link`.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; validation behavior; boundaries; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/link`, callers in `internal/config`, `internal/state`, `internal/publish`.
|
|
||||||
- Acceptance criteria: concise; URL rules match tests; link construction remains outside this package.
|
|
||||||
|
|
||||||
### `docs/internal/notify.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: notification interface and no-op implementation.
|
|
||||||
- Canonical scope: implemented notification behavior only.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; current behavior; boundaries; failure behavior; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/notify`, `internal/app/run_notify.go`, `internal/app/run_test.go`.
|
|
||||||
- Acceptance criteria: does not document external notifier config/adapters as implemented.
|
|
||||||
|
|
||||||
### `docs/internal/publish.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: publish planning/execution contract and safety behavior.
|
|
||||||
- Canonical scope: `internal/publish`.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; actions; output planning; link planning; transfer policy handling; safety/deletion; execution order; failure behavior; boundaries; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/publish/*.go`, `internal/publish/*_test.go`, `internal/state`, `internal/storage`, `internal/app/run_destination.go`.
|
|
||||||
- Acceptance criteria: force behavior and managed deletion are exact; app-owned selection/path mapping boundary is clear; no CLI/config reference duplication beyond naming consumed policies.
|
|
||||||
|
|
||||||
### `docs/internal/state.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: destination state schema, parsing, validation, and comparison internals.
|
|
||||||
- Canonical scope: `internal/state`.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; state schema; output records; link metadata; comparison outcomes; failure behavior; boundaries; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/state`, state tests, publish/state callers.
|
|
||||||
- Acceptance criteria: `.distributor.json` remains destination state canonical; comparison does not imply mutation; URL/output validation matches code.
|
|
||||||
|
|
||||||
### `docs/internal/storage.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: storage interface, logical paths, typed errors, traversal, deletion boundaries, fake backend.
|
|
||||||
- Canonical scope: `internal/storage` and backend interface semantics.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; boundaries; logical paths; errors; traversal helpers; deletion helpers; fake backend; tests; invariants.
|
|
||||||
- Source-of-truth repository areas: `internal/storage`, `internal/storage/fake`, adapter packages for interface compliance.
|
|
||||||
- Acceptance criteria: keeps adapter protocol details brief and links to integration docs for SSH/SFTP and S3; path and deletion safety match tests.
|
|
||||||
|
|
||||||
### `docs/internal/transform.md`
|
|
||||||
|
|
||||||
- Audience: developers and LLM coding agents.
|
|
||||||
- Purpose: transform registry and Markdown transform boundary.
|
|
||||||
- Canonical scope: `internal/transform` and implemented Markdown transform.
|
|
||||||
- Recommended outline: purpose; inputs/outputs; registry; Markdown behavior summary; failure behavior; boundaries; tests; invariants; link to `docs/integrations/markdown.md`.
|
|
||||||
- Source-of-truth repository areas: `internal/transform`, `internal/transform/markdown`, publish callers.
|
|
||||||
- Acceptance criteria: only Markdown-to-HTML is documented as implemented; generated output metadata matches code.
|
|
||||||
|
|
||||||
### `docs/integrations/source-bundle.md`
|
|
||||||
|
|
||||||
- Audience: producer developers and distributor maintainers.
|
|
||||||
- Purpose: external producer-facing source bundle and `manifest.json` contract.
|
|
||||||
- Canonical scope: implemented source bundle file format and `pkg/bundle` helper API overview.
|
|
||||||
- Recommended outline: source bundle layout; manifest schema; path/digest/timestamp rules; reserved paths; local producer helper package; validation command; examples.
|
|
||||||
- Source-of-truth repository areas: `pkg/bundle`, `internal/bundle`, `examples/source-bundle`, `docs/policy/architecture.md` source bundle contract.
|
|
||||||
- Acceptance criteria: no destination config or publish policy; schema matches `pkg/bundle`; links to CLI validate and examples.
|
|
||||||
|
|
||||||
### `docs/integrations/destination-state.md`
|
|
||||||
|
|
||||||
- Audience: operators, destination consumers, maintainers.
|
|
||||||
- Purpose: external destination `.distributor.json` state file contract.
|
|
||||||
- Canonical scope: implemented destination state JSON format and compatibility expectations.
|
|
||||||
- Recommended outline: file purpose; schema fields; embedded source manifest; outputs; links; comparison-relevant fields; safe manual inspection guidance.
|
|
||||||
- Source-of-truth repository areas: `internal/state`, `internal/publish/output.go`, `docs/policy/architecture.md` destination state contract.
|
|
||||||
- Acceptance criteria: does not instruct users to hand-edit state as normal workflow; describes implemented fields only.
|
|
||||||
|
|
||||||
### `docs/integrations/http-upload.md`
|
|
||||||
|
|
||||||
- Audience: producer developers and operators integrating with the upload API.
|
|
||||||
- Purpose: HTTP upload API contract.
|
|
||||||
- Canonical scope: implemented HTTP routes, auth, content types, responses, status lifecycle, and error boundary.
|
|
||||||
- Recommended outline: configured token-to-pipeline mapping; routes; auth header; accepted content types; archive requirements; response shapes; status lifecycle; retention; security notes.
|
|
||||||
- Source-of-truth repository areas: `internal/app/upload_http.go`, `internal/app/upload_coordinator.go`, `internal/ingest`, upload tests, `examples/http-upload-local.yml`.
|
|
||||||
- Acceptance criteria: documents staging/validation before run id; no durable queue claim; no TLS/rate-limit/browser UI claims.
|
|
||||||
|
|
||||||
### `docs/integrations/markdown.md`
|
|
||||||
|
|
||||||
- Audience: maintainers of transform behavior.
|
|
||||||
- Purpose: Goldmark integration contract.
|
|
||||||
- Canonical scope: implemented Markdown rendering behavior and dependency notes.
|
|
||||||
- Recommended outline: dependency; renderer options; raw HTML behavior; sidecar/index mapping; compatibility notes; tests.
|
|
||||||
- Source-of-truth repository areas: `go.mod`, `internal/transform/markdown`, markdown tests.
|
|
||||||
- Acceptance criteria: describes only behavior this project configures or depends on.
|
|
||||||
|
|
||||||
### `docs/integrations/ssh-sftp.md`
|
|
||||||
|
|
||||||
- Audience: operators and maintainers integrating SSH/SFTP storage.
|
|
||||||
- Purpose: SSH/SFTP backend contract and operational compatibility notes.
|
|
||||||
- Canonical scope: implemented SSH/SFTP protocol behavior, auth sources, host key policy, deletion safety.
|
|
||||||
- Recommended outline: libraries/protocol; config fields link; auth order; host key policies; known hosts behavior; path root; dry-run host key behavior; tests/integration gating.
|
|
||||||
- Source-of-truth repository areas: `internal/adapters/ssh`, `internal/app/backends.go`, `internal/config/ssh.go`, SSH adapter tests.
|
|
||||||
- Acceptance criteria: no password auth claim; no shell/rsync behavior; documents opt-in integration tests only if present in tests.
|
|
||||||
|
|
||||||
### `docs/integrations/s3.md`
|
|
||||||
|
|
||||||
- Audience: operators and maintainers integrating S3-compatible storage.
|
|
||||||
- Purpose: S3-compatible backend contract and operational compatibility notes.
|
|
||||||
- Canonical scope: implemented AWS SDK usage, endpoint/bucket/prefix behavior, credentials, object semantics, deletion safety.
|
|
||||||
- Recommended outline: dependency; endpoint/region/force path style; credentials; prefix/object key semantics; overwrite and content type behavior; tests/integration gating.
|
|
||||||
- Source-of-truth repository areas: `internal/adapters/s3`, `internal/app/backends.go`, `internal/config/s3.go`, S3 adapter tests.
|
|
||||||
- Acceptance criteria: no unsupported cloud-specific features; explicit credential resolver boundary is correct.
|
|
||||||
|
|
||||||
### `examples/`
|
|
||||||
|
|
||||||
- Audience: users, administrators, operators, developers.
|
|
||||||
- Purpose: copyable, valid configs and a source bundle fixture.
|
|
||||||
- Canonical scope: maintained examples for implemented behavior.
|
|
||||||
- Recommended outline: no separate README required unless examples become hard to navigate; link from README/config/CLI/operations.
|
|
||||||
- Source-of-truth repository areas: `examples/*.yml`, `examples/source-bundle`, `internal/config/load_test.go`.
|
|
||||||
- Acceptance criteria: all example YAML files load in tests; local examples can be run without external services; remote examples are clearly environment-gated and contain no secrets.
|
|
||||||
|
|
||||||
### `docs/roadmap/`
|
|
||||||
|
|
||||||
- Audience: maintainers, developers, LLM coding agents.
|
|
||||||
- Purpose: future/deferred work and implementation roadmaps only.
|
|
||||||
- Canonical scope: not current behavior.
|
|
||||||
- Recommended final contents: keep `docs/roadmap/documentation.md` until migration completes; keep any roadmap docs that still describe deferred work; delete completed roadmaps once their behavior is documented in current docs.
|
|
||||||
- Source-of-truth repository areas: current roadmap files plus current implementation.
|
|
||||||
- Acceptance criteria: no completed behavior is documented only in roadmap files; active roadmap files clearly distinguish future/deferred work from current behavior.
|
|
||||||
|
|
||||||
## File-by-File Rewrite Guidance
|
|
||||||
|
|
||||||
### `README.md`
|
|
||||||
|
|
||||||
Cover the project purpose, the shortest local run command, and links to canonical docs. Avoid full CLI/config details, implementation history, and feature roadmap summaries. Inspect `examples/local-publish.yml`, `internal/cli/root.go`, and `docs/cli.md` after rewrite. Do not carry forward any wording that suggests roadmap docs are required to understand implemented behavior.
|
|
||||||
|
|
||||||
### `docs/cli.md`
|
|
||||||
|
|
||||||
Cover the exact commands implemented in `internal/cli`: `version`, `run`, `serve`, `validate`, `inspect`, and `manifest create`. Include `--format text|json`, `--config`, `--dry-run`, `--force`, source diagnostic flags, and manifest creation flags. Explicitly include both manifest positional forms: `manifest create <path> --id x` and `manifest create --id x <path>`. Avoid duplicating full YAML reference or operations recovery procedures. Link to `docs/config.md` for fields and `docs/operations.md` for safety/recovery. Inspect `internal/cli/*.go` and `internal/cli/root_test.go` before writing.
|
|
||||||
|
|
||||||
### `docs/config.md`
|
|
||||||
|
|
||||||
Rewrite from the config structs, defaults, validation, and load tests. Cover all implemented fields, defaults, accepted values, source/destination backend differences, HTTP upload source-only behavior, S3 credentials, SSH host key policy, path mapping, links, publish/transform, transfer policy, secrets, and examples. Avoid internal package implementation details except where needed to explain user-visible behavior. Link to integration docs for protocol-level details. Inspect `internal/config/config.go`, `internal/config/defaults.go`, `internal/config/validate.go`, `internal/config/secrets.go`, `internal/app/backends.go`, and `internal/config/load_test.go`.
|
|
||||||
|
|
||||||
### `docs/operations.md`
|
|
||||||
|
|
||||||
Rewrite as an operations guide. Cover validation-before-run, dry-run, normal publish, fixed/latest destination safety, destination state, retry behavior, forced replacement, HTTP upload server lifecycle, memory-only status, and remote backend operational caveats. Avoid listing every config field or every CLI flag. Link to CLI/config/troubleshooting and integration docs. Inspect app run/upload tests, publish safety tests, storage deletion tests, and adapter tests.
|
|
||||||
|
|
||||||
### `docs/troubleshooting.md`
|
|
||||||
|
|
||||||
Rewrite around symptoms and safe fixes. Keep entries concise and link out. Cover config parsing/validation, source manifest failures, reserved/unsafe paths, digest mismatches, destination unmanaged/conflict/newer state, force requirements, SSH host key/auth issues, S3 credential/connectivity issues, HTTP upload `401`/`413`/`415`/`503`/`400`/`404`, and JSON partial results. Avoid broad tutorials and avoid exposing secret values. Inspect tests where errors are asserted.
|
|
||||||
|
|
||||||
### `docs/internal/*.md`
|
|
||||||
|
|
||||||
Normalize each internal doc to the policy-required component structure. Keep boundaries explicit: CLI parsing in `internal/cli`, orchestration in `internal/app`, config in `internal/config`, source manifests in `pkg/bundle`/`internal/bundle`, destination state in `internal/state`, storage interface in `internal/storage`, adapter protocol behavior in adapters, transforms in `internal/transform`, publication in `internal/publish`, upload staging in `internal/ingest`, URL policy in `internal/link`, notification interface in `internal/notify`. Avoid user-facing how-to content except links to canonical user docs. Do not document unimplemented notification adapters or future backends.
|
|
||||||
|
|
||||||
### `docs/integrations/*.md`
|
|
||||||
|
|
||||||
Keep integration docs concise and contract-focused. Create new entries only for implemented integrations. Do not copy full external documentation. Document only how this project uses the integration and which code/tests define compatibility. Link from config, operations, and internal docs where useful.
|
|
||||||
|
|
||||||
### `docs/roadmap/*.md`
|
|
||||||
|
|
||||||
After current docs are rewritten, review existing roadmap files. Delete or replace completed roadmap documents when they no longer describe future/deferred work. Preserve genuinely deferred work under `docs/roadmap/`. Avoid using roadmap files as changelogs.
|
|
||||||
|
|
||||||
## Examples Plan
|
|
||||||
|
|
||||||
`examples/` exists and should remain the canonical home for maintained examples.
|
|
||||||
|
|
||||||
Recommended examples:
|
|
||||||
|
|
||||||
- `examples/source-bundle/` — purpose: minimal valid source bundle fixture. Validity check: `go run ./cmd/distributor validate examples/source-bundle` or existing bundle tests. Link from README, CLI, source bundle integration doc.
|
|
||||||
- `examples/local-to-local.yml` — purpose: minimal local config shape. Validity check: `go test ./internal/config`; optional dry-run after replacing absolute paths with local temp paths is not directly runnable as-is. Link from config reference as a minimal production-shaped example.
|
|
||||||
- `examples/local-publish.yml` — purpose: runnable local source publication. Validity check: `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`; already load-tested. Link from README, CLI, operations.
|
|
||||||
- `examples/local-html.yml` — purpose: runnable sidecar Markdown-to-HTML publication. Validity check: dry-run command; load-tested. Link from config HTML section and CLI workflows.
|
|
||||||
- `examples/local-index.yml` — purpose: runnable `index.html` Markdown publication. Validity check: dry-run command; load-tested. Link from config HTML section.
|
|
||||||
- `examples/fan-out.yml` — purpose: one source to multiple local destinations. Validity check: dry-run command; load-tested. Link from operations and architecture/pipeline model if useful.
|
|
||||||
- `examples/archive-and-latest.yml` — purpose: archive plus fixed latest destination. Validity check: dry-run command; load-tested. Link from operations fixed-path safety and config path mapping.
|
|
||||||
- `examples/http-upload-local.yml` — purpose: local HTTP upload server with token environment reference. Validity check: `go test ./internal/config`; optional manual `serve` with `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN`. Link from CLI, config HTTP upload, operations, HTTP upload integration doc.
|
|
||||||
- `examples/ssh-destination.yml` — purpose: environment-gated SSH/SFTP destination shape. Validity check: `go test ./internal/config`; live execution only after replacing endpoint/path/key fields. Link from config SSH and SSH integration doc.
|
|
||||||
- `examples/s3-destination.yml` — purpose: environment-gated S3-compatible destination shape. Validity check: `go test ./internal/config`; live execution only after replacing endpoint/bucket/credentials. Link from config S3 and S3 integration doc.
|
|
||||||
|
|
||||||
Do not invent examples for unimplemented notification adapters, durable queues, TLS, zstd uploads, browser UI, or non-Markdown transforms.
|
|
||||||
|
|
||||||
## Internal Documentation Plan
|
|
||||||
|
|
||||||
Recommended internal component docs for implemented components:
|
|
||||||
|
|
||||||
- Component: app orchestration. Path: `docs/internal/app.md`. Purpose: top-level use cases, runtime setup, run report, upload coordination. Inputs/outputs: app option structs, config, writers, reports, HTTP requests/status records. Boundaries: no config schema ownership, no adapter internals, no manifest rules. Config fields used: config path, server HTTP, source/destination backend fields through runtime setup. Adapters used: local, SSH/SFTP, S3 through backend factory. Failure behavior: fatal setup errors, partial destination failures, upload admission/staging/status errors. Tests: `internal/app/*_test.go`, `internal/cli/root_test.go`. Invariants: fan-out independence, same run path after selection, dry-run safety, no secret leakage.
|
|
||||||
- Component: config. Path: `docs/internal/config.md`. Purpose: YAML structs, defaults, validation, secret resolver. Inputs/outputs: YAML path to `Config`, `Environment`, validation errors. Boundaries: no backend opening or upload execution. Config fields used: all config fields. Adapters used: none directly. Failure behavior: load/parse/validation/secret errors. Tests: `internal/config/*_test.go`, `internal/app/runtime_test.go`. Invariants: known fields only, defaults before validation, source-only HTTP upload.
|
|
||||||
- Component: source bundles. Path: `docs/internal/bundle.md`. Purpose: storage-backed discovery/validation and public manifest contract delegation. Inputs/outputs: storage backend trees to validated bundle list. Boundaries: concrete adapters hidden behind storage; producer API in `pkg/bundle`. Config fields used: source root indirectly. Adapters used: storage interface only. Failure behavior: manifest parse/validation, file stat/read/digest failures. Tests: `internal/bundle`, `pkg/bundle`. Invariants: one root manifest per bundle, nested manifests rejected, path safety.
|
|
||||||
- Component: ingestion. Path: `docs/internal/ingest.md`. Purpose: archive content-type validation, extraction, source bundle staging. Inputs/outputs: upload body/content type/limits/staging path/run id to committed local bundle root. Boundaries: no auth, queueing, HTTP routing, or publish. Config fields used: effective staging path and upload size supplied by app. Adapters used: local filesystem only. Failure behavior: unsupported content type, oversize, unsafe tar entries, validation failure, cleanup. Tests: `internal/ingest/archive_test.go`. Invariants: invalid archives cannot commit staged roots.
|
|
||||||
- Component: publish. Path: `docs/internal/publish.md`. Purpose: plan/execute destination actions and safety. Inputs/outputs: source bundle/backend, destination backend/state/policies to plan and writes. Boundaries: no CLI parsing/config loading/source selection. Config fields used: publish, transform, transfer, links after app/config validation. Adapters used: storage interface and transform resolver. Failure behavior: conflicts, unmanaged content, output collisions, write/delete failures. Tests: `internal/publish`, `internal/app/run_test.go`. Invariants: state after outputs, bounded deletion, force explicit.
|
|
||||||
- Component: state. Path: `docs/internal/state.md`. Purpose: `.distributor.json` schema and comparison. Inputs/outputs: JSON/current source identity to comparison outcome. Boundaries: no storage mutation. Config fields used: none directly. Adapters used: none. Failure behavior: invalid JSON/schema/URLs/output paths/conflicts. Tests: `internal/state`. Invariants: state is sentinel, comparison pure, embedded manifests validated.
|
|
||||||
- Component: storage. Path: `docs/internal/storage.md`. Purpose: storage interface, logical path policy, errors, traversal, deletion helpers, fake backend. Inputs/outputs: logical paths and backend operations. Boundaries: adapters own protocol specifics. Config fields used: none directly. Adapters used: local/SSH/S3 implement interface; fake for tests. Failure behavior: typed storage errors, path validation, traversal stop. Tests: `internal/storage`, `internal/storage/fake`, adapter tests. Invariants: slash paths, root confinement, bounded deletion.
|
|
||||||
- Component: transform. Path: `docs/internal/transform.md`. Purpose: transform interface/registry and Markdown generation summary. Inputs/outputs: validated source bundle/source backend/options to generated output records. Boundaries: no publication or state writing. Config fields used: transform policy supplied by publish. Adapters used: source storage backend for reads. Failure behavior: unresolved transform, invalid index input, read/render errors. Tests: `internal/transform`, `internal/transform/markdown`. Invariants: source immutability, deterministic output metadata.
|
|
||||||
- Component: link. Path: `docs/internal/link.md`. Purpose: shared HTTP URL validation. Inputs/outputs: URL string to validation result. Boundaries: no URL construction. Config fields used: `links.base_url` via config. Adapters used: none. Failure behavior: parse/scheme/host/query/fragment validation. Tests: `internal/link`, callers. Invariants: same URL policy for config and persisted state.
|
|
||||||
- Component: notify. Path: `docs/internal/notify.md`. Purpose: notification interface and no-op behavior. Inputs/outputs: notification event to error. Boundaries: no configured external notifications. Config fields used: none currently. Adapters used: none. Failure behavior: no-op returns context cancellation only. Tests: `internal/notify`, app notify tests. Invariants: only successful publish/replacement notifies; dry-run never notifies.
|
|
||||||
|
|
||||||
Only recommend internal docs for implemented components. Do not add docs for nonexistent workflow engines, notification adapters, durable queues, or non-Markdown transforms.
|
|
||||||
|
|
||||||
## Integration Documentation Plan
|
|
||||||
|
|
||||||
Recommended integration docs for implemented external contracts:
|
|
||||||
|
|
||||||
- Path: `docs/integrations/source-bundle.md`. External system or contract: producer-created source bundle directory and `manifest.json` schema. Current usage: producer-facing `pkg/bundle`, CLI `manifest create`, app/source validation. Version notes: `schema_version: 1`; digest format `sha256:<64 lowercase hex>`; RFC3339 timestamps. Document schema, validation, reserved paths, examples, and `pkg/bundle` helper role. Do not document destination routing or config policies as part of source manifests.
|
|
||||||
- Path: `docs/integrations/destination-state.md`. External system or contract: destination `.distributor.json` state file. Current usage: destination sentinel, comparison state, output and link metadata. Version notes: `schema_version: 1`; `distributor_version` diagnostic. Document fields, output kinds, embedded manifest, links, comparison role. Do not recommend routine hand-editing or future state fields.
|
|
||||||
- Path: `docs/integrations/http-upload.md`. External system or contract: HTTP upload API. Current usage: `serve` routes `/healthz`, `/upload`, `/runs/<run_id>`, bearer token auth, tar/tar.gz archive ingestion. Version notes: no explicit API version; response schema is implemented in app tests. Document routes, request/response shapes, status values, content types, queue/status retention. Do not document TLS, rate limiting, durable queues, idempotency, zstd, or browser UI as implemented.
|
|
||||||
- Path: `docs/integrations/markdown.md`. External system or contract: Goldmark Markdown renderer. Current usage: Markdown-to-HTML transform. Version notes: dependency version is in `go.mod`; document project-configured behavior instead of the full Goldmark API. Document raw HTML behavior, sidecar/index modes, deterministic output, tests. Do not document unsupported Markdown extensions unless configured in code.
|
|
||||||
- Path: `docs/integrations/ssh-sftp.md`. External system or contract: SSH/SFTP storage. Current usage: source and destination backend adapter using native SSH/SFTP libraries. Version notes: dependency versions in `go.mod`; live integration tests are opt-in. Document auth order, host key policies, known_hosts behavior, path root, dry-run host key persistence behavior. Do not document password auth, shell commands, rsync, SCP, or unsupported SSH features.
|
|
||||||
- Path: `docs/integrations/s3.md`. External system or contract: S3-compatible object storage. Current usage: source and destination backend adapter using AWS SDK for Go v2. Version notes: dependency versions in `go.mod`; supports configured endpoint, bucket, prefix, region, force path style. Document credential resolution, prefix semantics, traversal/stat/write/delete behavior, content type inference, live test gating. Do not document cloud-provider-specific features not used by the adapter.
|
|
||||||
|
|
||||||
No external CLI integrations were found. The project uses Go libraries and protocols directly.
|
|
||||||
|
|
||||||
## Recommended Implementation Sequence
|
|
||||||
|
|
||||||
### Stage 1: Documentation Structure And Roadmap Cleanup
|
|
||||||
|
|
||||||
- Goal: establish the target documentation tree and remove or quarantine completed roadmap noise without rewriting user docs yet.
|
|
||||||
- Files to create/update/delete/move: update `docs/roadmap/documentation.md` only if needed; decide which completed roadmap files remain, but delete them only in a later implementation pass after current docs are rewritten.
|
|
||||||
- Repository areas to inspect: `docs/roadmap/*`, `docs/policy/documentation.md`, current docs links.
|
|
||||||
- Acceptance criteria: roadmap docs contain only future/deferred/planning material; no current behavior is documented only in old roadmap files.
|
|
||||||
- Suggested validation commands: `find docs -maxdepth 3 -type f | sort`; `rg -n -i "future|planned|deferred|experimental|deprecated|not implemented" README.md docs --glob '!docs/roadmap/**'`.
|
|
||||||
- Prompt size: small enough for one implementation prompt.
|
|
||||||
|
|
||||||
### Stage 2: README And Canonical User Entry Points
|
|
||||||
|
|
||||||
- Goal: rewrite `README.md` and `docs/cli.md` as concise, accurate user entry points.
|
|
||||||
- Files to create/update/delete/move: `README.md`, `docs/cli.md`.
|
|
||||||
- Repository areas to inspect: `internal/cli`, `internal/cli/root_test.go`, `internal/app` output code, examples.
|
|
||||||
- Acceptance criteria: README is concise; CLI flags and examples match parser code/tests; manifest positional parsing examples are included; no full config reference duplication.
|
|
||||||
- Suggested validation commands: `go test ./internal/cli ./internal/app`; manual `go run ./cmd/distributor <command> --help` spot checks.
|
|
||||||
- Prompt size: small enough for one implementation prompt.
|
|
||||||
|
|
||||||
### Stage 3: Configuration Reference And Examples Links
|
|
||||||
|
|
||||||
- Goal: rewrite the canonical config reference and verify examples list.
|
|
||||||
- Files to create/update/delete/move: `docs/config.md`; examples only if stale or invalid.
|
|
||||||
- Repository areas to inspect: `internal/config`, `internal/app/backends.go`, `internal/app/serve.go`, `examples`, config tests.
|
|
||||||
- Acceptance criteria: all fields/defaults/accepted values match code; examples list is complete; remote examples are clearly environment-gated; no secrets.
|
|
||||||
- Suggested validation commands: `go test ./internal/config`; `rg -n "examples/" docs/config.md README.md docs/cli.md docs/operations.md`.
|
|
||||||
- Prompt size: one implementation prompt if examples remain unchanged; split if examples require edits.
|
|
||||||
|
|
||||||
### Stage 4: Operations And Troubleshooting
|
|
||||||
|
|
||||||
- Goal: rewrite operator-facing workflow, safety, recovery, and symptom docs.
|
|
||||||
- Files to create/update/delete/move: `docs/operations.md`, `docs/troubleshooting.md`.
|
|
||||||
- Repository areas to inspect: `internal/app/run*.go`, `internal/app/upload*.go`, `internal/publish`, `internal/state`, `internal/storage`, adapter tests, CLI tests.
|
|
||||||
- Acceptance criteria: operations doc avoids full field/flag tables; troubleshooting entries use symptom/cause/diagnostic/safe fix/link; HTTP upload and forced replacement behavior match code.
|
|
||||||
- Suggested validation commands: `go test ./internal/app ./internal/publish ./internal/state ./internal/storage`; stale-term grep outside roadmap.
|
|
||||||
- Prompt size: likely too large for one prompt if both docs are long; split into operations first, troubleshooting second.
|
|
||||||
|
|
||||||
### Stage 5: Integration Contracts
|
|
||||||
|
|
||||||
- Goal: create concise integration docs for implemented external contracts.
|
|
||||||
- Files to create/update/delete/move: create `docs/integrations/source-bundle.md`, `docs/integrations/destination-state.md`, `docs/integrations/http-upload.md`, `docs/integrations/ssh-sftp.md`, `docs/integrations/s3.md`; update `docs/integrations/markdown.md`.
|
|
||||||
- Repository areas to inspect: `pkg/bundle`, `internal/state`, `internal/app/upload_http.go`, `internal/ingest`, `internal/adapters/ssh`, `internal/adapters/s3`, `internal/transform/markdown`, `go.mod`, tests.
|
|
||||||
- Acceptance criteria: integration docs cover only implemented contracts; no external docs copied wholesale; current docs link to integration docs where appropriate.
|
|
||||||
- Suggested validation commands: `go test ./pkg/bundle ./internal/state ./internal/app ./internal/ingest ./internal/adapters/ssh ./internal/adapters/s3 ./internal/transform/markdown`.
|
|
||||||
- Prompt size: split into source/state/http and markdown/SSH/S3 if needed.
|
|
||||||
|
|
||||||
### Stage 6: Internal Component Docs Normalization
|
|
||||||
|
|
||||||
- Goal: normalize `docs/internal/` to policy-required sections and current component boundaries.
|
|
||||||
- Files to create/update/delete/move: all `docs/internal/*.md`.
|
|
||||||
- Repository areas to inspect: matching internal package code/tests and integration docs from Stage 5.
|
|
||||||
- Acceptance criteria: each internal doc includes purpose, inputs/outputs, boundaries, config fields used, adapters used, failure behavior, tests, and invariants where applicable; user-facing how-to content is linked instead of duplicated.
|
|
||||||
- Suggested validation commands: package-specific `go test` for touched components; `rg -n "future|planned|deferred|not implemented" docs/internal`.
|
|
||||||
- Prompt size: too large for one prompt; split by package group: app/config/ingest, bundle/state/publish, storage/adapters/transform/link/notify.
|
|
||||||
|
|
||||||
### Stage 7: Final Roadmap And Link Hygiene
|
|
||||||
|
|
||||||
- Goal: remove completed roadmap files that no longer represent future work and check links/stale terminology.
|
|
||||||
- Files to create/update/delete/move: `docs/roadmap/*` as appropriate; no code.
|
|
||||||
- Repository areas to inspect: all docs, README, examples.
|
|
||||||
- Acceptance criteria: completed behavior is documented in current docs, not only roadmaps; no stale completed roadmap instructions remain as active plans; links are accurate.
|
|
||||||
- Suggested validation commands: `find docs -maxdepth 3 -type f | sort`; `rg -n -i "future|planned|deferred|experimental|deprecated|old behavior|Stage|Phase" README.md docs --glob '!docs/roadmap/**'`; manual link review.
|
|
||||||
- Prompt size: small enough for one implementation prompt after prior rewrites.
|
|
||||||
|
|
||||||
### Stage 8: Full Documentation Validation
|
|
||||||
|
|
||||||
- Goal: verify docs against tests, examples, and policy checklist.
|
|
||||||
- Files to create/update/delete/move: fixes only if validation finds gaps.
|
|
||||||
- Repository areas to inspect: full repo.
|
|
||||||
- Acceptance criteria: tests pass; examples load; CLI examples match parser; stale-term grep passes; documentation checklist in `docs/policy/documentation.md` is satisfied.
|
|
||||||
- Suggested validation commands: `go test ./...`; `go test ./internal/config ./internal/cli ./internal/app`; stale-term and link greps listed below.
|
|
||||||
- Prompt size: small enough for one implementation prompt if prior stages are complete.
|
|
||||||
|
|
||||||
## Validation Plan
|
|
||||||
|
|
||||||
Recommended validation checks during and after documentation implementation:
|
|
||||||
|
|
||||||
- Full test suite: `go test ./...`.
|
|
||||||
- Config/example loading: `go test ./internal/config` because `internal/config/load_test.go` loads maintained examples.
|
|
||||||
- CLI/parser behavior: `go test ./internal/cli ./internal/app`.
|
|
||||||
- App/report/upload behavior: `go test ./internal/app ./internal/ingest`.
|
|
||||||
- Publish/state/storage safety: `go test ./internal/publish ./internal/state ./internal/storage ./internal/storage/fake`.
|
|
||||||
- Adapter docs: `go test ./internal/adapters/local ./internal/adapters/ssh ./internal/adapters/s3`.
|
|
||||||
- Transform docs: `go test ./internal/transform/markdown`.
|
|
||||||
- Producer/source bundle docs: `go test ./pkg/bundle ./internal/bundle`.
|
|
||||||
- Stale terminology outside roadmap:
|
|
||||||
```sh
|
|
||||||
rg -n -i "future|planned|deferred|experimental|deprecated|not implemented|old behavior" README.md docs --glob '!docs/roadmap/**'
|
|
||||||
```
|
|
||||||
- Stale completed-work labels outside roadmap:
|
|
||||||
```sh
|
|
||||||
rg -n -i "Stage|Phase" README.md docs --glob '!docs/roadmap/**'
|
|
||||||
```
|
|
||||||
- CLI flag consistency:
|
|
||||||
```sh
|
|
||||||
rg -n "--config|--dry-run|--force|--format|--pipeline|--bundle|--id|--file|--created|--overwrite" docs/cli.md internal/cli
|
|
||||||
```
|
|
||||||
- Example references:
|
|
||||||
```sh
|
|
||||||
rg -n "examples/" README.md docs examples internal/config/load_test.go
|
|
||||||
```
|
|
||||||
- Link checks: no automated Markdown link checker was found. Perform manual link review or add a link checker in a separate tooling change if desired.
|
|
||||||
|
|
||||||
Manual review items:
|
Manual review items:
|
||||||
|
|
||||||
@@ -569,11 +87,11 @@ Manual review items:
|
|||||||
- Confirm `docs/config.md` is the only full config field/default reference.
|
- Confirm `docs/config.md` is the only full config field/default reference.
|
||||||
- Confirm `docs/cli.md` is the only full command/flag reference.
|
- Confirm `docs/cli.md` is the only full command/flag reference.
|
||||||
- Confirm `docs/operations.md` focuses on operating and recovery.
|
- Confirm `docs/operations.md` focuses on operating and recovery.
|
||||||
- Confirm `docs/troubleshooting.md` is symptom-first.
|
- Confirm `docs/troubleshooting.md` remains symptom-first.
|
||||||
- Confirm `docs/internal/` docs describe implemented component contracts and boundaries.
|
- Confirm `docs/internal/` describes implemented component contracts and boundaries.
|
||||||
- Confirm integration docs do not claim support for unimplemented external features.
|
- Confirm integration docs do not claim support for unimplemented external features.
|
||||||
- Confirm examples contain no secrets and distinguish local runnable examples from environment-gated remote examples.
|
- Confirm examples contain no secrets and distinguish local runnable examples from environment-gated remote examples.
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
No open questions block this roadmap. The recommended approach is to rewrite the documentation from the current code and tests, preserve all implemented behavior in current docs, move only actual integration contracts into `docs/integrations/`, and delete completed roadmap documents only after their implemented behavior is represented in canonical current docs.
|
No open questions block the remaining validation work.
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ The HTTP upload API is implemented. Current behavior is documented in:
|
|||||||
- [Configuration](../config.md)
|
- [Configuration](../config.md)
|
||||||
- [Operations](../operations.md)
|
- [Operations](../operations.md)
|
||||||
- [Troubleshooting](../troubleshooting.md)
|
- [Troubleshooting](../troubleshooting.md)
|
||||||
|
- [HTTP upload contract](../integrations/http-upload.md)
|
||||||
- [Application internals](../internal/app.md)
|
- [Application internals](../internal/app.md)
|
||||||
- [Ingestion internals](../internal/ingest.md)
|
- [Ingestion internals](../internal/ingest.md)
|
||||||
|
|
||||||
|
|||||||
@@ -1,34 +0,0 @@
|
|||||||
# HTTP Upload Deferred Work
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
HTTP upload behavior is implemented and documented in the current-behavior
|
|
||||||
manuals:
|
|
||||||
|
|
||||||
- [CLI](../cli.md)
|
|
||||||
- [Configuration](../config.md)
|
|
||||||
- [Operations](../operations.md)
|
|
||||||
- [Troubleshooting](../troubleshooting.md)
|
|
||||||
- [Application internals](../internal/app.md)
|
|
||||||
- [Configuration internals](../internal/config.md)
|
|
||||||
- [Ingestion internals](../internal/ingest.md)
|
|
||||||
|
|
||||||
This file tracks only HTTP upload work that is not implemented.
|
|
||||||
|
|
||||||
## Deferred Work
|
|
||||||
|
|
||||||
- URL-token authentication.
|
|
||||||
- Zstandard-compressed archive support.
|
|
||||||
- Durable status persistence across process restarts.
|
|
||||||
- Database-backed queueing.
|
|
||||||
- Producer-supplied idempotency keys.
|
|
||||||
- Run listing, cancellation, and retry endpoints.
|
|
||||||
- In-app TLS.
|
|
||||||
- Public network exposure defaults.
|
|
||||||
- Browser UI.
|
|
||||||
|
|
||||||
## Documentation Rule
|
|
||||||
|
|
||||||
Deferred behavior belongs under `docs/roadmap/` until implemented. Current
|
|
||||||
behavior docs must describe only the active HTTP upload API, configuration,
|
|
||||||
operation, troubleshooting, and internal package contracts.
|
|
||||||
Reference in New Issue
Block a user