Compare commits
4 Commits
v0.6.0
...
3da7f931b2
| Author | SHA1 | Date | |
|---|---|---|---|
| 3da7f931b2 | |||
| b10a8bd194 | |||
| c84d8868d1 | |||
| fc33bbca54 |
@@ -313,6 +313,7 @@ transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
css_href: /assets/report.css
|
||||
```
|
||||
|
||||
`publish.html` controls whether generated HTML outputs are published. When `publish.html` is `true`, `transform.markdown_to_html.enabled` must also be `true`.
|
||||
@@ -322,10 +323,13 @@ Markdown transform fields:
|
||||
- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for this destination.
|
||||
- `transform.markdown_to_html.mode`: optional. Accepted values are `sidecar` and `index`; default is `sidecar` when a Markdown transform block is present.
|
||||
- `transform.markdown_to_html.input`: optional source manifest path for `index` mode only.
|
||||
- `transform.markdown_to_html.css_href`: optional stylesheet href to link from generated HTML.
|
||||
|
||||
`sidecar` mode renders every manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one Markdown source to `index.html` at the destination bundle path. If `index` mode omits `input`, the selected source bundle must contain exactly one Markdown file.
|
||||
|
||||
At least one output type must be enabled. Enabled Markdown transforms are rejected when `publish.html` is `false`, and `input` is rejected unless `mode` is `index`.
|
||||
`css_href` may be an absolute `http` or `https` URL, a root-relative path such as `/assets/report.css`, or a relative URL path such as `assets/report.css`. Query strings are allowed. `distributor` injects the href as a `<link rel="stylesheet">` element but does not copy, publish, verify, or manage the CSS file solely because `css_href` is set.
|
||||
|
||||
At least one output type must be enabled. Enabled Markdown transforms are rejected when `publish.html` is `false`, `input` is rejected unless `mode` is `index`, and `css_href` is rejected when the Markdown transform is disabled.
|
||||
|
||||
## Destination Path Mapping
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Rendering uses `github.com/yuin/goldmark`. The exact version is pinned in `go.mo
|
||||
|
||||
## Renderer Behavior
|
||||
|
||||
The transformer constructs `goldmark.New()` with no project-specific extensions, parser options, renderer options, templates, CSS, or metadata injection.
|
||||
The transformer constructs `goldmark.New()` with no project-specific extensions, parser options, renderer options, templates, or source manifest metadata injection.
|
||||
|
||||
Supported output modes:
|
||||
|
||||
@@ -19,6 +19,8 @@ Supported output modes:
|
||||
|
||||
In `index` mode, `transform.markdown_to_html.input` may name the source manifest path to render. If `input` is omitted, the source manifest must list exactly one `.md` file. The selected input must be a clean relative source path, must be listed in the source manifest, and must end in `.md`.
|
||||
|
||||
When `transform.markdown_to_html.css_href` is set, generated HTML includes a stylesheet link in the document head. The href may be an absolute HTTP(S) URL, a root-relative path, or a relative URL path. Distributor treats this as a link reference only; it does not copy, publish, verify, or manage the CSS file solely because `css_href` is configured.
|
||||
|
||||
Raw HTML embedded in Markdown is not passed through by the current renderer behavior. Tests allow Goldmark's disabled-or-escaped raw HTML output forms and reject literal script tags in generated HTML.
|
||||
|
||||
## HTML Wrapper
|
||||
@@ -28,10 +30,11 @@ Rendered Markdown body HTML is wrapped in a fixed document shell:
|
||||
- `<!doctype html>`
|
||||
- `<html lang="en">`
|
||||
- UTF-8 `<meta charset>`
|
||||
- optional `<link rel="stylesheet" href="...">` when `css_href` is configured
|
||||
- empty `<title>`
|
||||
- `<body>` containing the rendered Markdown body
|
||||
|
||||
The wrapper is deterministic and does not read configuration, templates, CSS, or source manifest metadata.
|
||||
The wrapper is deterministic. When `css_href` is omitted, the generated wrapper is unchanged from the unstyled output. When `css_href` is configured, its escaped link element is part of the generated output bytes.
|
||||
|
||||
## Output Metadata
|
||||
|
||||
|
||||
@@ -43,6 +43,7 @@ Canonical homes:
|
||||
|
||||
- project purpose and quickstart: `README.md`
|
||||
- development principles: `docs/policy/architecture.md`
|
||||
- public HTTP API reference: `docs/api.md`
|
||||
- configuration reference: `docs/config.md`
|
||||
- CLI reference: `docs/cli.md`
|
||||
- operations and recovery: `docs/operations.md`
|
||||
@@ -122,6 +123,22 @@ Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- validated examples under `examples/`
|
||||
|
||||
### Public HTTP API service
|
||||
|
||||
Required:
|
||||
- `docs/api.md`
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/consumers/`, for task-oriented client integration guides
|
||||
- `docs/integrations/`, for upstream/downstream service contracts
|
||||
- validated examples under `examples/`
|
||||
|
||||
### Project with public packages or consumer APIs
|
||||
|
||||
Required:
|
||||
@@ -173,6 +190,32 @@ It should include:
|
||||
|
||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||
|
||||
### docs/api.md
|
||||
|
||||
**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
|
||||
|
||||
Required for projects whose primary public interface is HTTP.
|
||||
|
||||
`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
|
||||
|
||||
It should include:
|
||||
|
||||
1. base URL conventions;
|
||||
2. authentication and authorization behavior, if implemented;
|
||||
3. response envelope;
|
||||
4. supported media types and content negotiation behavior;
|
||||
5. shared query parameters;
|
||||
6. endpoint reference grouped by route family;
|
||||
7. request parameters and validation rules;
|
||||
8. response fields, units, nullability, and optionality;
|
||||
9. error response shape and status codes;
|
||||
10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
|
||||
11. compact request and response examples.
|
||||
|
||||
It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
|
||||
|
||||
For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
|
||||
|
||||
### docs/policy/development.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
@@ -264,6 +307,8 @@ Required for projects with public packages, SDKs, client APIs, plugin APIs, or o
|
||||
|
||||
This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
|
||||
|
||||
For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
|
||||
|
||||
`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
|
||||
|
||||
1. intended consumer audience and use cases;
|
||||
@@ -330,6 +375,8 @@ Required for projects that depend on external CLIs, APIs, services, protocols, o
|
||||
|
||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
|
||||
|
||||
For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
|
||||
|
||||
Use one file per integration where useful.
|
||||
|
||||
## Examples Directory
|
||||
@@ -385,9 +432,10 @@ Before merging documentation changes, verify:
|
||||
|
||||
- README is concise and orientation-focused.
|
||||
- `docs/policy/architecture.md` describes development principles.
|
||||
- `docs/api.md` is the canonical HTTP contract for HTTP API services.
|
||||
- Future work appears only under `docs/roadmap/`.
|
||||
- User-facing docs avoid unnecessary internals.
|
||||
- Consumer-facing docs explain public APIs without duplicating integration contracts.
|
||||
- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
|
||||
- Developer-facing docs preserve boundaries and invariants.
|
||||
- Config examples match the schema.
|
||||
- CLI examples match real commands and flags.
|
||||
|
||||
375
docs/roadmap/implementation.md
Normal file
375
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,375 @@
|
||||
# Managed Destination Takeover Implementation Roadmap
|
||||
|
||||
This is the active staged implementation plan for
|
||||
`docs/roadmap/takeover.md`. The feature roadmap defines the target policy and
|
||||
end-state semantics; this document defines the implementation sequence for an
|
||||
LLM coding agent to follow stage by stage.
|
||||
|
||||
Future behavior must remain under `docs/roadmap/` until each stage is
|
||||
implemented. Preparatory internal stages should not update user-facing current
|
||||
docs. Current-behavior docs should be updated when takeover behavior is wired
|
||||
for operator-facing use.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
`distributor` already supports local, SSH/SFTP, S3, and HTTP upload source
|
||||
workflows, destination state schemas for single-owner and shared-root state,
|
||||
path mapping, link generation, reconciliation, transfer policy, explicit
|
||||
`--force`, and text/JSON run output.
|
||||
|
||||
Current destination comparison is strict:
|
||||
|
||||
- same source manifest skips;
|
||||
- same source id with older destination state normally replaces;
|
||||
- same source id with newer destination state normally skips;
|
||||
- same source id and same creation time with different digest conflicts;
|
||||
- different source id, pipeline id, destination id, or shared-root output owner
|
||||
conflicts unless explicit force policy applies;
|
||||
- unmanaged content and invalid state do not become normal managed replacement
|
||||
cases.
|
||||
|
||||
The target feature adds destination-level `takeover.mode`, defaulting to
|
||||
`same_pipeline`, so valid managed content can be replaced without `--force` when
|
||||
the configured policy says that ownership/source takeover is expected.
|
||||
|
||||
## Implementation Principles
|
||||
|
||||
- Preserve public behavior until the stage that explicitly changes it.
|
||||
- Keep source manifests unchanged; takeover is destination configuration and
|
||||
publish planning policy.
|
||||
- Keep state comparison pure. State comparison may expose structured conflict
|
||||
details, but publish planning decides whether takeover is allowed.
|
||||
- Keep adapters thin. No local, SSH, or S3 adapter should know takeover policy.
|
||||
- Keep unmanaged content and invalid destination state outside normal takeover.
|
||||
- Keep `reconciliation.mode`, `transfer`, `state.mode`, `path_mapping.mode`, and
|
||||
`--force` as separate concepts.
|
||||
- Prefer narrow behavior-preserving refactors over broad publication rewrites.
|
||||
- Update implemented-behavior docs in the same stage as the behavior change.
|
||||
|
||||
## Active Implementation Stages
|
||||
|
||||
## Stage 1: Config Model And Validation
|
||||
|
||||
Goal: add the destination config field, defaults, and validation without
|
||||
changing publish behavior.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- `docs/roadmap/takeover.md`: Configuration, Policy Semantics, Safety Rules.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add destination-level `TakeoverPolicy` to `internal/config`.
|
||||
- Add constants for:
|
||||
- `same_pipeline`
|
||||
- `same_source`
|
||||
- `any_managed`
|
||||
- `never`
|
||||
- Default `takeover.mode` to `same_pipeline` in config defaults.
|
||||
- Validate accepted values with clear field context such as
|
||||
`pipelines[0].destinations[0].takeover.mode`.
|
||||
- Preserve strict YAML unknown-field behavior.
|
||||
- Thread the defaulted policy into existing destination config views or helper
|
||||
structures if those are used by app/publish request construction.
|
||||
- Do not change publish planning, execution, CLI output, or docs outside
|
||||
roadmap files in this stage.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/config`
|
||||
- Add config tests for omitted `takeover`, each accepted mode, invalid mode, and
|
||||
unknown nested fields.
|
||||
- Add or update example-loading tests only if examples are touched, which should
|
||||
not be necessary in this stage.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Every destination has a defaulted `takeover.mode`.
|
||||
- Invalid values fail during config validation.
|
||||
- No run behavior changes because publish planning does not consume the policy
|
||||
yet.
|
||||
|
||||
## Stage 2: Structured State Comparison Details
|
||||
|
||||
Goal: expose enough structured comparison detail for publish planning to decide
|
||||
takeover eligibility without parsing human-readable reason strings.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- `docs/roadmap/takeover.md`: Policy Semantics, Relationship To Existing
|
||||
Policies.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Extend `internal/state.Comparison` or add a package-local structured detail
|
||||
type so callers can distinguish:
|
||||
- pipeline id mismatch;
|
||||
- destination id mismatch;
|
||||
- different source id;
|
||||
- same-created digest conflict;
|
||||
- destination newer;
|
||||
- invalid state;
|
||||
- unmanaged content;
|
||||
- absent shared-root owner;
|
||||
- shared-root managed output owner conflicts, if those are currently reported
|
||||
outside `internal/state`.
|
||||
- Keep existing outcome names and reason strings stable where practical.
|
||||
- Keep comparison functions pure. They should report facts about existing state,
|
||||
not consult `takeover.mode`, `transfer`, `reconciliation`, or `force`.
|
||||
- Do not add new persisted state fields.
|
||||
- Do not change publish actions in this stage.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/publish`
|
||||
- Add state tests for structured details on identity mismatch and different
|
||||
source id.
|
||||
- Add shared-root tests for owner absence and output ownership conflict detail,
|
||||
either in `internal/state` or `internal/publish` depending on where the
|
||||
conflict is currently detected.
|
||||
- Preserve existing comparison outcome tests.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Publish planning can make takeover decisions from structured data.
|
||||
- Existing behavior remains unchanged because no takeover mapping is applied yet.
|
||||
|
||||
## Stage 3: Single-Owner Takeover Planning And Execution
|
||||
|
||||
Goal: implement `takeover.mode` for single-owner destination state.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- `docs/roadmap/takeover.md`: Policy Semantics, Publish Planning, Destination
|
||||
State Results, Safety Rules.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add `TakeoverPolicy` or equivalent values to `internal/publish.Request`.
|
||||
- Pass destination takeover policy from app-level run planning into publish.
|
||||
- Add publish action `replace_takeover`.
|
||||
- Map eligible single-owner conflicts to `replace_takeover`:
|
||||
- `same_pipeline`: existing state pipeline id equals current pipeline id,
|
||||
regardless of destination id or source id;
|
||||
- `same_source`: existing state source id equals current source id;
|
||||
- `any_managed`: existing state is valid distributor-managed state;
|
||||
- `never`: no identity/source takeover.
|
||||
- Keep these cases failing by default unless existing force behavior applies:
|
||||
- unmanaged content;
|
||||
- invalid state;
|
||||
- same-created digest conflict;
|
||||
- destination newer for the same source id unless `transfer` plus `--force`
|
||||
already permits replacement;
|
||||
- conflicts not allowed by `takeover.mode`.
|
||||
- Execute `replace_takeover` through bounded managed replacement mechanics.
|
||||
- For cross-source takeover, do not retain omitted outputs through
|
||||
`reconciliation.mode: merge`; treat the affected single-owner state as a
|
||||
managed replacement so old-source outputs are not attributed to the new source.
|
||||
- Preserve existing `replace_older`, `skip_same`, `skip_destination_newer`,
|
||||
`fail_conflict`, `fail_unmanaged`, and `force_replace` behavior.
|
||||
- Keep adapters unchanged.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
|
||||
- Do not update user docs yet unless CLI output changes in this stage. Prefer
|
||||
deferring user docs to Stage 5 so the behavior, output, and docs land
|
||||
together.
|
||||
- If this stage changes visible dry-run or run output enough that tests require
|
||||
new wording, document only the implemented single-owner behavior and clearly
|
||||
leave shared-root takeover out until Stage 4.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/publish ./internal/app`
|
||||
- Default `same_pipeline` replaces a different source id from the same pipeline.
|
||||
- Default `same_pipeline` replaces state with a different destination id under
|
||||
the same pipeline.
|
||||
- Default `same_pipeline` refuses a different pipeline.
|
||||
- `same_source` allows same source id and refuses different source id.
|
||||
- `any_managed` replaces valid state from a different pipeline.
|
||||
- `never` refuses identity/source takeover.
|
||||
- Invalid state and unmanaged content still fail without force.
|
||||
- Cross-source takeover with `reconciliation.mode: merge` does not retain
|
||||
omitted outputs from the previous source.
|
||||
- Dry-run plans `replace_takeover` without writing.
|
||||
- Existing force tests still pass.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Single-owner latest-style destinations can be updated by different bundle ids
|
||||
from the same configured pipeline without `--force`.
|
||||
- No unmanaged or invalid-state path becomes a normal takeover path.
|
||||
|
||||
## Stage 4: Shared-Root Takeover Planning And Execution
|
||||
|
||||
Goal: apply the same takeover vocabulary to shared-root owner and output-path
|
||||
conflicts.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- `docs/roadmap/takeover.md`: Policy Semantics, Publish Planning, Destination
|
||||
State Results, Safety Rules.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Extend shared-root planning so planned output collisions with existing managed
|
||||
owners are eligible for takeover according to `takeover.mode`.
|
||||
- Apply policy as follows:
|
||||
- `same_pipeline`: current owner may take over output paths owned by another
|
||||
destination under the same pipeline;
|
||||
- `same_source`: current owner may take over output paths whose owner records
|
||||
the same source id as the current source;
|
||||
- `any_managed`: current owner may take over output paths owned by any valid
|
||||
shared-root owner;
|
||||
- `never`: preserve current owner conflict behavior.
|
||||
- Preserve unrelated owner records and non-conflicting output records.
|
||||
- Keep unmanaged path collisions failing without force.
|
||||
- Keep compatible single-owner-to-shared-root migration behavior intact.
|
||||
- For cross-source takeover, do not retain omitted outputs from the previous
|
||||
source under the taking-over owner when `reconciliation.mode: merge` is set.
|
||||
- Keep shared-root forced replacement behavior explicit and bounded as it is
|
||||
today.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
|
||||
- Defer broad documentation updates to Stage 5 unless shared-root behavior must
|
||||
be documented immediately to keep tests or generated docs consistent.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/publish ./internal/app`
|
||||
- `same_pipeline` permits taking over a path owned by another destination in the
|
||||
same pipeline.
|
||||
- `same_pipeline` refuses a path owned by another pipeline.
|
||||
- `same_source` permits only matching source-id ownership transfer.
|
||||
- `any_managed` permits cross-pipeline managed ownership transfer.
|
||||
- `never` preserves current shared-root conflict behavior.
|
||||
- Reconciliation `replace` and `merge` handle omitted outputs according to the
|
||||
feature roadmap, including the cross-source no-retain rule.
|
||||
- Shared-root migration from compatible single-owner state still works.
|
||||
- Unmanaged collisions still fail without force.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Single-owner and shared-root destinations use one consistent takeover policy.
|
||||
- Shared-root takeover changes only affected owners/outputs and preserves
|
||||
unrelated managed state.
|
||||
|
||||
## Stage 5: Run Output, Summaries, And Documentation
|
||||
|
||||
Goal: make takeover behavior visible to operators and document the implemented
|
||||
feature.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- `docs/roadmap/takeover.md`: Documentation Impact, Publish Planning,
|
||||
Relationship To Existing Policies.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add `replace_takeover` to text run output.
|
||||
- Add `replace_takeover` to JSON run action output.
|
||||
- Add a distinct summary counter for takeover replacements in text and JSON run
|
||||
summaries, using the field name `replace_takeover` for JSON.
|
||||
- Include takeover mode and conflict reason in action projection where useful
|
||||
and consistent with existing output style.
|
||||
- Update fixed-path dry-run warnings when takeover will replace the destination
|
||||
root.
|
||||
- Update current-behavior docs:
|
||||
- `docs/config.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/integrations/destination-state.md`
|
||||
- `docs/internal/publish.md`
|
||||
- `docs/internal/state.md`
|
||||
- `docs/internal/config.md`
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/development.md`
|
||||
- Keep docs concise and link to canonical references rather than duplicating
|
||||
full state semantics in every file.
|
||||
- Do not document any unimplemented future takeover extensions outside
|
||||
`docs/roadmap/`.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/app ./internal/cli ./internal/config`
|
||||
- Text dry-run output includes `replace_takeover`.
|
||||
- JSON output records `replace_takeover` with stable field names.
|
||||
- Summary counters are deterministic.
|
||||
- Fixed-path warnings remain deterministic.
|
||||
- Config docs examples, if changed or added, load through existing config tests.
|
||||
- Troubleshooting text matches actual error/action wording.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Operators can distinguish ordinary same-source replacement, takeover
|
||||
replacement, and explicit forced replacement.
|
||||
- Current docs describe implemented takeover behavior outside roadmap files.
|
||||
|
||||
## Stage 6: Cross-Backend Regression And Roadmap Closeout
|
||||
|
||||
Goal: prove takeover behavior is backend-agnostic and clean up roadmap state
|
||||
after implementation.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- `docs/roadmap/takeover.md`: Goals, Non-Goals, Safety Rules.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add app-level coverage showing takeover planning/execution works with storage
|
||||
abstraction rather than adapter-specific logic.
|
||||
- Cover local destinations directly.
|
||||
- Cover SSH and S3 through fake-backed app tests where possible; add adapter
|
||||
integration tests only if existing test infrastructure already supports
|
||||
opt-in remote credentials.
|
||||
- Add tests for `path_mapping.mode: fixed` with different source ids under the
|
||||
same pipeline.
|
||||
- Add tests for archive-style `preserve_relative` destinations using
|
||||
`takeover.mode: same_source` where strict source identity is desired.
|
||||
- Run final consistency searches.
|
||||
- Once behavior and docs are complete, either remove `docs/roadmap/takeover.md`
|
||||
or reduce it to future-only material according to the documentation policy.
|
||||
If no future takeover work remains, remove the roadmap file.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/config`
|
||||
- `go test ./internal/state`
|
||||
- `go test ./internal/publish`
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- `go test ./...`
|
||||
|
||||
Documentation and consistency checks:
|
||||
|
||||
```sh
|
||||
rg -n "takeover|replace_takeover|same_pipeline|same_source|any_managed" docs README.md examples
|
||||
rg -n "destination source id differs from source|pipeline id .* does not match|destination id .* does not match" docs
|
||||
rg -n "Managed Destination Takeover Roadmap|future takeover|planned takeover" docs README.md examples --glob '!docs/roadmap/**'
|
||||
```
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Full test suite passes.
|
||||
- The default `same_pipeline` behavior applies only to valid
|
||||
distributor-managed takeover cases.
|
||||
- Unmanaged content and invalid state remain protected.
|
||||
- Current docs no longer describe implemented takeover behavior as future work.
|
||||
|
||||
## Refactors To Avoid
|
||||
|
||||
- Do not build a generic ownership engine outside `internal/state` and
|
||||
`internal/publish`.
|
||||
- Do not move destination comparison policy into storage adapters.
|
||||
- Do not change the source manifest schema.
|
||||
- Do not add takeover fields to HTTP upload requests or producer APIs.
|
||||
- Do not make `--force` persistent config.
|
||||
- Do not merge `takeover`, `transfer`, and `reconciliation` into one broad
|
||||
policy object.
|
||||
- Do not silently adopt unmanaged content.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No open questions are known. The feature roadmap selects the default mode
|
||||
(`same_pipeline`), the accepted modes, the safety boundary, and the relationship
|
||||
to existing state, reconciliation, transfer, and force policies.
|
||||
229
docs/roadmap/takeover.md
Normal file
229
docs/roadmap/takeover.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# Managed Destination Takeover Roadmap
|
||||
|
||||
This roadmap defines planned replacement policy for distributor-managed
|
||||
destination content when the existing destination state does not describe the
|
||||
same source bundle as the current publication.
|
||||
|
||||
Current behavior is intentionally conservative. A destination may be replaced
|
||||
automatically when its existing state describes the same source id and an older
|
||||
source creation time. If the existing state belongs to a different source id,
|
||||
pipeline id, destination id, or shared-root owner, `distributor` reports a
|
||||
conflict unless an explicit forced replacement workflow is selected where
|
||||
supported.
|
||||
|
||||
That strict default was useful for the initial implementation, but it is too
|
||||
rigid for producers that intentionally publish newer generated content to the
|
||||
same destination path from different producer tasks or bundle ids. `distributor`
|
||||
is primarily a distribution tool: by default, a valid producer request should
|
||||
publish the requested bundle unless doing so would claim unmanaged content or
|
||||
cross an ownership boundary that is likely to surprise the operator.
|
||||
|
||||
## Goals
|
||||
|
||||
- Add a destination-level policy that defines when normal, non-force publication
|
||||
may replace existing distributor-managed content whose source or owner
|
||||
identity differs from the current source.
|
||||
- Default the policy to `same_pipeline`, so a pipeline may distribute the latest
|
||||
valid bundle to its configured destination paths without requiring every
|
||||
producer task to reuse one source manifest id.
|
||||
- Preserve strong safety boundaries for unmanaged content, invalid destination
|
||||
state, and cross-pipeline ownership by default.
|
||||
- Keep reconciliation policy focused on how managed output sets are updated
|
||||
after replacement is allowed.
|
||||
- Keep transfer policy focused on same-source comparison outcomes and explicit
|
||||
forced replacement behavior.
|
||||
- Preserve clear dry-run and JSON output so takeover behavior is visible before
|
||||
files are changed.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not allow automatic adoption of unmanaged destination files.
|
||||
- Do not allow invalid `.distributor.json` state to be overwritten without the
|
||||
existing explicit force workflow.
|
||||
- Do not make producers select destination ownership or takeover policy through
|
||||
source manifests or HTTP upload requests.
|
||||
- Do not replace `state.mode`, `reconciliation.mode`, `path_mapping.mode`, or
|
||||
`transfer` policy.
|
||||
- Do not introduce broad synchronization behavior outside configured
|
||||
destination bundle paths.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add destination-level takeover policy:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: latest
|
||||
backend: local
|
||||
path: /srv/reports/weather/latest/tomorrow
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
takeover:
|
||||
mode: same_pipeline
|
||||
```
|
||||
|
||||
Accepted `takeover.mode` values:
|
||||
|
||||
- `same_pipeline`: default. Normal publication may replace valid
|
||||
distributor-managed state or managed output paths owned by the same pipeline,
|
||||
even when the previous source id or destination id differs.
|
||||
- `same_source`: normal publication may replace only when the destination state
|
||||
has the same source manifest id as the current source. This is closest to the
|
||||
current strict managed replacement behavior.
|
||||
- `any_managed`: normal publication may replace any valid distributor-managed
|
||||
state or managed output path in the selected destination bundle path,
|
||||
regardless of pipeline id, destination id, or source id.
|
||||
- `never`: normal publication never performs takeover replacement for identity
|
||||
or source conflicts. Same-source older/newer behavior remains governed by
|
||||
`transfer`, and explicit forced replacement remains governed by `--force` and
|
||||
`transfer.on_conflict`.
|
||||
|
||||
The field is destination-local because different destinations from the same
|
||||
pipeline can have different ownership expectations. Archive destinations often
|
||||
want stricter behavior, while latest-style fixed destinations often want
|
||||
`same_pipeline` or, after explicit operator review, `any_managed`.
|
||||
|
||||
## Policy Semantics
|
||||
|
||||
Takeover policy applies only after destination state has been parsed and
|
||||
validated as distributor-managed state.
|
||||
|
||||
For single-owner state:
|
||||
|
||||
- `same_pipeline` permits replacement when the existing state `pipeline_id`
|
||||
matches the current pipeline id. The previous `destination_id` and source id
|
||||
may differ.
|
||||
- `same_source` permits replacement only when the existing source manifest id
|
||||
matches the current source manifest id.
|
||||
- `any_managed` permits replacement of any valid single-owner destination state.
|
||||
- `never` does not permit takeover replacement.
|
||||
|
||||
For shared-root state:
|
||||
|
||||
- Takeover is evaluated per managed output path that collides with the current
|
||||
plan and per current owner record when one already exists.
|
||||
- `same_pipeline` permits the current destination owner to take over output
|
||||
paths owned by another destination under the same pipeline.
|
||||
- `same_source` permits takeover only when the colliding output owner records
|
||||
the same source manifest id as the current source.
|
||||
- `any_managed` permits takeover of colliding managed output paths owned by any
|
||||
owner in the shared root.
|
||||
- `never` preserves the current owner-scoped conflict behavior.
|
||||
|
||||
Takeover never applies to:
|
||||
|
||||
- destination content with no valid `.distributor.json`;
|
||||
- invalid destination state;
|
||||
- planned paths that exist in storage but are not recorded as managed;
|
||||
- source digest mismatches or invalid source manifests;
|
||||
- destination newer/same-created digest comparisons for the same source id,
|
||||
except where existing transfer policy already permits replacement.
|
||||
|
||||
## Relationship To Existing Policies
|
||||
|
||||
`takeover.mode` answers this question:
|
||||
|
||||
> May this destination normally replace valid managed content whose owner or
|
||||
> source identity differs from the current publication?
|
||||
|
||||
`reconciliation.mode` continues to answer:
|
||||
|
||||
> Once replacement is allowed, should omitted managed outputs be removed
|
||||
> (`replace`) or retained (`merge`)?
|
||||
|
||||
`transfer` continues to answer:
|
||||
|
||||
> What should happen for same-source comparisons, destination-newer state, and
|
||||
> explicit forced conflict replacement?
|
||||
|
||||
`--force` remains an operator workflow for exceptional cases. Takeover policy is
|
||||
not a replacement for forced replacement of unmanaged content or invalid state.
|
||||
|
||||
## Publish Planning
|
||||
|
||||
Add a new planned action for successful takeover replacement, tentatively named
|
||||
`replace_takeover`.
|
||||
|
||||
`replace_takeover` should execute through the same bounded managed replacement
|
||||
machinery as other managed replacements, with one additional source-identity
|
||||
safety rule: when takeover crosses source id or owner identity, omitted outputs
|
||||
from the previous source must not be retained merely because the destination is
|
||||
configured with `reconciliation.mode: merge`. Retaining omitted outputs is valid
|
||||
only when those outputs still belong to the same source identity being
|
||||
published. This keeps destination state from claiming that outputs derived from
|
||||
an old source belong to the new source manifest.
|
||||
|
||||
- single-owner `replace` deletes only managed outputs recorded in existing
|
||||
state plus `.distributor.json`, then writes the new outputs and state;
|
||||
- single-owner `merge` retains omitted managed outputs only for same-source
|
||||
updates; cross-source takeover behaves as a managed replacement for the
|
||||
affected owner;
|
||||
- shared-root `replace` deletes only output records taken over by the current
|
||||
owner or omitted from the current owner according to the shared-root plan;
|
||||
- shared-root `merge` preserves unrelated non-conflicting managed outputs and
|
||||
updates ownership for paths explicitly taken over, but does not retain omitted
|
||||
outputs from a different source under the taking-over owner.
|
||||
|
||||
Text output, JSON output, dry-run output, and summary counters should identify
|
||||
takeover replacements separately from `replace_older` and `force_replace`.
|
||||
|
||||
## Destination State Results
|
||||
|
||||
After a successful takeover, destination state must describe the current
|
||||
pipeline, destination, source manifest, outputs, links, reconciliation policy,
|
||||
and state mode.
|
||||
|
||||
For single-owner state, the destination bundle path has one current owner after
|
||||
publication: the publishing pipeline and destination.
|
||||
|
||||
For shared-root state, only the affected owner records and output records are
|
||||
changed. Unrelated owners and non-conflicting outputs remain recorded.
|
||||
|
||||
No source manifest schema change is required. No destination state schema change
|
||||
is expected unless implementation discovers that takeover metadata must be
|
||||
persisted for recovery or audit. The preferred initial design is to make
|
||||
takeover a planning decision, not a new persisted state concept.
|
||||
|
||||
## Documentation Impact
|
||||
|
||||
When this feature is implemented, current-behavior docs should explain the new
|
||||
policy where operators and maintainers already look for publication safety,
|
||||
state comparison, and configuration behavior:
|
||||
|
||||
- `docs/config.md`: document `takeover.mode`, default `same_pipeline`, accepted
|
||||
values, examples for archive and latest destinations, and interaction with
|
||||
`transfer`, `reconciliation`, and `--force`.
|
||||
- `docs/operations.md`: explain takeover dry-runs, normal replacement safety,
|
||||
shared-root behavior, and recovery guidance.
|
||||
- `docs/troubleshooting.md`: update conflict guidance for
|
||||
`destination source id differs from source`, pipeline/destination mismatch,
|
||||
and shared-root ownership conflicts.
|
||||
- `docs/integrations/destination-state.md`: describe how takeover affects
|
||||
comparison and state rewriting without changing the source manifest contract.
|
||||
- `docs/internal/publish.md`, `docs/internal/state.md`, and
|
||||
`docs/internal/config.md`: document package responsibilities and invariants.
|
||||
- `docs/policy/architecture.md`: clarify that the default posture is to
|
||||
distribute valid producer bundles through configured destinations while
|
||||
protecting unmanaged content and cross-pipeline ownership by default.
|
||||
- `docs/policy/development.md`: add `takeover.mode` to the list of config
|
||||
changes that must be validated and documented when publish safety changes.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- Takeover must be dry-runnable.
|
||||
- Takeover must operate only inside the resolved destination bundle path.
|
||||
- Takeover must never delete parent paths, sibling paths, or unmanaged files.
|
||||
- Takeover must never claim storage content that is not recorded in valid
|
||||
destination state.
|
||||
- `any_managed` should be documented as an intentional cross-pipeline ownership
|
||||
policy for tightly controlled destinations, not as the recommended default for
|
||||
archives.
|
||||
- Archive-style destinations should generally use `same_source` if each bundle
|
||||
id represents a distinct immutable artifact.
|
||||
|
||||
## Implementation Reference
|
||||
|
||||
The staged implementation plan for this feature lives in
|
||||
`docs/roadmap/implementation.md`. This document remains the feature contract and
|
||||
policy reference; implementation sequencing, package-by-package work, and test
|
||||
commands belong in the implementation roadmap.
|
||||
@@ -170,7 +170,7 @@ func (b *Backend) WriteFrom(ctx context.Context, logicalPath string, r io.Reader
|
||||
return storage.Entry{}, storage.NewError(storage.OpWriteFrom, BackendName, logicalPath, storage.ErrConflict, fmt.Errorf("stream size %d does not match expected size %d", written, opts.Size))
|
||||
}
|
||||
if opts.PreferAtomic {
|
||||
if err := b.client.Rename(writePath, nativePath); err != nil {
|
||||
if err := renamePromotedFile(b.client, writePath, nativePath, opts.Overwrite); err != nil {
|
||||
return storage.Entry{}, b.translateError(storage.OpWriteFrom, logicalPath, err)
|
||||
}
|
||||
cleanup = false
|
||||
@@ -178,6 +178,27 @@ func (b *Backend) WriteFrom(ctx context.Context, logicalPath string, r io.Reader
|
||||
return b.Stat(ctx, logicalPath)
|
||||
}
|
||||
|
||||
type sftpRenamer interface {
|
||||
PosixRename(oldname, newname string) error
|
||||
Rename(oldname, newname string) error
|
||||
Remove(path string) error
|
||||
}
|
||||
|
||||
func renamePromotedFile(client sftpRenamer, oldname, newname string, overwrite bool) error {
|
||||
if !overwrite {
|
||||
return client.Rename(oldname, newname)
|
||||
}
|
||||
if err := client.PosixRename(oldname, newname); err == nil {
|
||||
return nil
|
||||
} else if !isReplaceRenameFallbackError(err) {
|
||||
return err
|
||||
}
|
||||
if err := client.Remove(newname); err != nil && !isNotExist(err) {
|
||||
return err
|
||||
}
|
||||
return client.Rename(oldname, newname)
|
||||
}
|
||||
|
||||
func (b *Backend) Stat(ctx context.Context, logicalPath string) (storage.Entry, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return storage.Entry{}, err
|
||||
@@ -464,6 +485,14 @@ func isNotExist(err error) bool {
|
||||
return errors.Is(err, fs.ErrNotExist) || errors.Is(err, os.ErrNotExist) || errors.Is(err, sftp.ErrSSHFxNoSuchFile)
|
||||
}
|
||||
|
||||
func isReplaceRenameFallbackError(err error) bool {
|
||||
if errors.Is(err, sftp.ErrSSHFxFailure) || errors.Is(err, sftp.ErrSSHFxOpUnsupported) {
|
||||
return true
|
||||
}
|
||||
var statusErr *sftp.StatusError
|
||||
return errors.As(err, &statusErr) && (statusErr.FxCode() == sftp.ErrSSHFxFailure || statusErr.FxCode() == sftp.ErrSSHFxOpUnsupported)
|
||||
}
|
||||
|
||||
func (b *Backend) translateError(op, logicalPath string, err error) error {
|
||||
kind := storage.ErrUnknown
|
||||
switch {
|
||||
|
||||
125
internal/adapters/ssh/backend_test.go
Normal file
125
internal/adapters/ssh/backend_test.go
Normal file
@@ -0,0 +1,125 @@
|
||||
package ssh
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/pkg/sftp"
|
||||
)
|
||||
|
||||
func TestRenamePromotedFileUsesPlainRenameWithoutOverwrite(t *testing.T) {
|
||||
client := &recordingRenamer{}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", false); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
if got, want := client.calls, []string{"rename temp index.html"}; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileUsesPosixRenameForOverwrite(t *testing.T) {
|
||||
client := &recordingRenamer{}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
if got, want := client.calls, []string{"posix temp index.html"}; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileFallsBackWhenReplaceRenameUnsupported(t *testing.T) {
|
||||
for _, err := range []error{
|
||||
sftp.ErrSSHFxOpUnsupported,
|
||||
sftp.ErrSSHFxFailure,
|
||||
&sftp.StatusError{Code: uint32(sftp.ErrSSHFxOpUnsupported)},
|
||||
&sftp.StatusError{Code: uint32(sftp.ErrSSHFxFailure)},
|
||||
} {
|
||||
t.Run(err.Error(), func(t *testing.T) {
|
||||
client := &recordingRenamer{posixErr: err}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
want := []string{"posix temp index.html", "remove index.html", "rename temp index.html"}
|
||||
if got := client.calls; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileIgnoresMissingTargetDuringFallback(t *testing.T) {
|
||||
client := &recordingRenamer{
|
||||
posixErr: sftp.ErrSSHFxOpUnsupported,
|
||||
removeErr: &os.PathError{
|
||||
Op: "remove",
|
||||
Path: "index.html",
|
||||
Err: os.ErrNotExist,
|
||||
},
|
||||
}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); err != nil {
|
||||
t.Fatalf("renamePromotedFile() error = %v", err)
|
||||
}
|
||||
want := []string{"posix temp index.html", "remove index.html", "rename temp index.html"}
|
||||
if got := client.calls; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileDoesNotFallbackForPermissionError(t *testing.T) {
|
||||
client := &recordingRenamer{posixErr: sftp.ErrSSHFxPermissionDenied}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); !errors.Is(err, sftp.ErrSSHFxPermissionDenied) {
|
||||
t.Fatalf("renamePromotedFile() error = %v, want permission denied", err)
|
||||
}
|
||||
if got, want := client.calls, []string{"posix temp index.html"}; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenamePromotedFileReturnsRemoveFallbackError(t *testing.T) {
|
||||
client := &recordingRenamer{
|
||||
posixErr: sftp.ErrSSHFxOpUnsupported,
|
||||
removeErr: sftp.ErrSSHFxPermissionDenied,
|
||||
}
|
||||
if err := renamePromotedFile(client, "temp", "index.html", true); !errors.Is(err, sftp.ErrSSHFxPermissionDenied) {
|
||||
t.Fatalf("renamePromotedFile() error = %v, want permission denied", err)
|
||||
}
|
||||
want := []string{"posix temp index.html", "remove index.html"}
|
||||
if got := client.calls; !equalStrings(got, want) {
|
||||
t.Fatalf("calls = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
type recordingRenamer struct {
|
||||
calls []string
|
||||
posixErr error
|
||||
renameErr error
|
||||
removeErr error
|
||||
}
|
||||
|
||||
func (r *recordingRenamer) PosixRename(oldname, newname string) error {
|
||||
r.calls = append(r.calls, "posix "+oldname+" "+newname)
|
||||
return r.posixErr
|
||||
}
|
||||
|
||||
func (r *recordingRenamer) Rename(oldname, newname string) error {
|
||||
r.calls = append(r.calls, "rename "+oldname+" "+newname)
|
||||
return r.renameErr
|
||||
}
|
||||
|
||||
func (r *recordingRenamer) Remove(path string) error {
|
||||
r.calls = append(r.calls, "remove "+path)
|
||||
return r.removeErr
|
||||
}
|
||||
|
||||
func equalStrings(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for index := range a {
|
||||
if a[index] != b[index] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -110,6 +110,7 @@ type MarkdownToHTML struct {
|
||||
Enabled bool `yaml:"enabled"`
|
||||
Mode string `yaml:"mode"`
|
||||
Input string `yaml:"input"`
|
||||
CssHref string `yaml:"css_href"`
|
||||
}
|
||||
|
||||
type PathMapping struct {
|
||||
|
||||
@@ -2,8 +2,10 @@ package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/link"
|
||||
)
|
||||
@@ -281,9 +283,15 @@ func ValidatePublishTransformPolicy(publish PublishPolicy, transform Transform)
|
||||
if transform.MarkdownToHTML.Input != "" && !transform.MarkdownToHTML.Enabled {
|
||||
return fmt.Errorf("transform.markdown_to_html.input requires transform.markdown_to_html.enabled to be true")
|
||||
}
|
||||
if transform.MarkdownToHTML.CssHref != "" && !transform.MarkdownToHTML.Enabled {
|
||||
return fmt.Errorf("transform.markdown_to_html.css_href requires transform.markdown_to_html.enabled to be true")
|
||||
}
|
||||
if transform.MarkdownToHTML.Input != "" && mode != TransformModeIndex {
|
||||
return fmt.Errorf("transform.markdown_to_html.input is only valid when mode is %s", TransformModeIndex)
|
||||
}
|
||||
if err := validateCSSHref(transform.MarkdownToHTML.CssHref); err != nil {
|
||||
return fmt.Errorf("transform.markdown_to_html.css_href %w", err)
|
||||
}
|
||||
if transform.MarkdownToHTML.Enabled && !publish.HTML {
|
||||
return fmt.Errorf("transform.markdown_to_html.enabled requires publish.html to be true")
|
||||
}
|
||||
@@ -293,6 +301,49 @@ func ValidatePublishTransformPolicy(publish PublishPolicy, transform Transform)
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateCSSHref(value string) error {
|
||||
if value == "" {
|
||||
return nil
|
||||
}
|
||||
for _, character := range value {
|
||||
if unicode.IsControl(character) || unicode.IsSpace(character) {
|
||||
return fmt.Errorf("must not contain whitespace or control characters")
|
||||
}
|
||||
}
|
||||
if strings.ContainsAny(value, "\\<>\"'") {
|
||||
return fmt.Errorf("must not contain backslashes or HTML-sensitive characters")
|
||||
}
|
||||
if strings.HasPrefix(value, "//") {
|
||||
return fmt.Errorf("must not be scheme-relative")
|
||||
}
|
||||
parsed, err := url.Parse(value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("must be a valid URL reference: %w", err)
|
||||
}
|
||||
if parsed.Fragment != "" {
|
||||
return fmt.Errorf("must not include a fragment")
|
||||
}
|
||||
if parsed.Scheme != "" {
|
||||
if parsed.Scheme != "http" && parsed.Scheme != "https" {
|
||||
return fmt.Errorf("scheme must be http or https")
|
||||
}
|
||||
if parsed.Host == "" {
|
||||
return fmt.Errorf("host is required for absolute URLs")
|
||||
}
|
||||
if parsed.User != nil {
|
||||
return fmt.Errorf("must not include userinfo")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if parsed.Host != "" {
|
||||
return fmt.Errorf("must not be scheme-relative")
|
||||
}
|
||||
if parsed.Path == "" {
|
||||
return fmt.Errorf("relative URL path is required")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validatePathMapping(errs ValidationErrors, context string, mapping PathMapping) ValidationErrors {
|
||||
if mapping.Mode != PathMappingPreserveRelative && mapping.Mode != PathMappingFixed {
|
||||
errs = append(errs, context+".mode must be "+PathMappingPreserveRelative+" or "+PathMappingFixed)
|
||||
|
||||
@@ -328,6 +328,24 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
|
||||
Input: "report.md",
|
||||
}},
|
||||
},
|
||||
{
|
||||
name: "html only sidecar css href allowed",
|
||||
publish: PublishPolicy{HTML: true},
|
||||
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
|
||||
Enabled: true,
|
||||
Mode: TransformModeSidecar,
|
||||
CssHref: "/assets/report.css",
|
||||
}},
|
||||
},
|
||||
{
|
||||
name: "html only index css href allowed",
|
||||
publish: PublishPolicy{HTML: true},
|
||||
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
|
||||
Enabled: true,
|
||||
Mode: TransformModeIndex,
|
||||
CssHref: "assets/report.css?v=20260614",
|
||||
}},
|
||||
},
|
||||
{
|
||||
name: "source and html sidecar allowed",
|
||||
publish: PublishPolicy{Source: true, HTML: true},
|
||||
@@ -425,6 +443,16 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
|
||||
}},
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "disabled markdown css href rejected",
|
||||
publish: PublishPolicy{Source: true},
|
||||
transform: Transform{MarkdownToHTML: &MarkdownToHTML{
|
||||
Enabled: false,
|
||||
Mode: TransformModeSidecar,
|
||||
CssHref: "/assets/report.css",
|
||||
}},
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "disabled markdown wrong mode rejected",
|
||||
publish: PublishPolicy{Source: true},
|
||||
@@ -436,3 +464,44 @@ func publishTransformPolicyCases() []publishTransformPolicyCase {
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateCSSHref(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
value string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "empty"},
|
||||
{name: "root relative", value: "/assets/report.css"},
|
||||
{name: "relative", value: "assets/report.css"},
|
||||
{name: "parent relative", value: "../assets/report.css"},
|
||||
{name: "query", value: "/assets/report.css?v=20260614"},
|
||||
{name: "http", value: "http://example.com/report.css"},
|
||||
{name: "https", value: "https://example.com/assets/report.css?v=1"},
|
||||
{name: "javascript", value: "javascript:alert(1)", wantErr: true},
|
||||
{name: "data", value: "data:text/css,body{}", wantErr: true},
|
||||
{name: "file", value: "file:///tmp/report.css", wantErr: true},
|
||||
{name: "scheme relative", value: "//example.com/report.css", wantErr: true},
|
||||
{name: "userinfo", value: "https://user@example.com/report.css", wantErr: true},
|
||||
{name: "fragment", value: "/assets/report.css#main", wantErr: true},
|
||||
{name: "space", value: "/assets/report css", wantErr: true},
|
||||
{name: "tab", value: "/assets/report\tcss", wantErr: true},
|
||||
{name: "newline", value: "/assets/report\ncss", wantErr: true},
|
||||
{name: "backslash", value: `assets\report.css`, wantErr: true},
|
||||
{name: "less than", value: "/assets/<report>.css", wantErr: true},
|
||||
{name: "double quote", value: `/assets/"report".css`, wantErr: true},
|
||||
{name: "single quote", value: "/assets/'report'.css", wantErr: true},
|
||||
{name: "query only", value: "?v=1", wantErr: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := validateCSSHref(tt.value)
|
||||
if tt.wantErr && err == nil {
|
||||
t.Fatal("validateCSSHref() error = nil, want error")
|
||||
}
|
||||
if !tt.wantErr && err != nil {
|
||||
t.Fatalf("validateCSSHref() error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -27,8 +27,9 @@ func PlanOutputs(ctx context.Context, req Request) ([]Output, error) {
|
||||
SourceBundle: req.SourceBundle,
|
||||
SourceBackend: req.SourceBackend,
|
||||
Markdown: transform.MarkdownOptions{
|
||||
Mode: req.Transform.MarkdownToHTML.Mode,
|
||||
Input: req.Transform.MarkdownToHTML.Input,
|
||||
Mode: req.Transform.MarkdownToHTML.Mode,
|
||||
Input: req.Transform.MarkdownToHTML.Input,
|
||||
CssHref: req.Transform.MarkdownToHTML.CssHref,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
|
||||
@@ -167,6 +167,7 @@ func TestPlanOutputsPassesMarkdownOptions(t *testing.T) {
|
||||
Enabled: true,
|
||||
Mode: config.TransformModeIndex,
|
||||
Input: "report.md",
|
||||
CssHref: "/assets/report.css",
|
||||
}},
|
||||
Transformers: testResolver{transform.MarkdownToHTML: transformer},
|
||||
})
|
||||
@@ -174,8 +175,8 @@ func TestPlanOutputsPassesMarkdownOptions(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("PlanOutputs() error = %v", err)
|
||||
}
|
||||
if transformer.request.Markdown.Mode != config.TransformModeIndex || transformer.request.Markdown.Input != "report.md" {
|
||||
t.Fatalf("markdown options = %#v, want index/report.md", transformer.request.Markdown)
|
||||
if transformer.request.Markdown.Mode != config.TransformModeIndex || transformer.request.Markdown.Input != "report.md" || transformer.request.Markdown.CssHref != "/assets/report.css" {
|
||||
t.Fatalf("markdown options = %#v, want index/report.md with css href", transformer.request.Markdown)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -123,7 +123,7 @@ func (t *Transformer) render(ctx context.Context, req transform.Request, sourceF
|
||||
if err := t.renderer.Convert(data, &rendered); err != nil {
|
||||
return nil, fmt.Errorf("render markdown source %q: %w", sourceFile, err)
|
||||
}
|
||||
return wrapHTML(rendered.Bytes()), nil
|
||||
return wrapHTML(rendered.Bytes(), req.Markdown.CssHref), nil
|
||||
}
|
||||
|
||||
func markdownMode(mode string) string {
|
||||
|
||||
@@ -40,6 +40,43 @@ func TestGenerateMarkdownSidecar(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownWithoutCSSHrefPreservesWrapper(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
want := "<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title></title>\n</head>\n<body>\n<h1>Title</h1>\n<p>Hello.</p>\n</body>\n</html>\n"
|
||||
if got := string(outputs[0].Data); got != want {
|
||||
t.Fatalf("html = %q, want existing wrapper %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownSidecarWithCSSHref(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
outputs, err := New().Generate(context.Background(), transform.Request{
|
||||
SourceBackend: backend,
|
||||
SourceBundle: sourceBundle,
|
||||
Markdown: transform.MarkdownOptions{CssHref: "/assets/report.css?v=1&theme=main"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
output := outputs[0]
|
||||
html := string(output.Data)
|
||||
wantLink := "<meta charset=\"utf-8\">\n<link rel=\"stylesheet\" href=\"/assets/report.css?v=1&theme=main\">\n<title></title>"
|
||||
if !strings.Contains(html, wantLink) {
|
||||
t.Fatalf("html = %q, want stylesheet link %q", html, wantLink)
|
||||
}
|
||||
if output.SHA256 != bundle.FileDigest(output.Data) || output.Size != int64(len(output.Data)) {
|
||||
t.Fatalf("digest/size metadata = %s/%d", output.SHA256, output.Size)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownIndexExplicitInput(t *testing.T) {
|
||||
backend := fake.New()
|
||||
sourceBundle := testutil.WriteFakeSourceBundle(t, backend, "", testutil.BundleOptions{
|
||||
@@ -74,6 +111,29 @@ func TestGenerateMarkdownIndexExplicitInput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownIndexWithCSSHref(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
outputs, err := New().Generate(context.Background(), transform.Request{
|
||||
SourceBackend: backend,
|
||||
SourceBundle: sourceBundle,
|
||||
Markdown: transform.MarkdownOptions{
|
||||
Mode: transform.MarkdownModeIndex,
|
||||
CssHref: "https://example.com/assets/report.css",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
if got, want := outputs[0].Path, "index.html"; got != want {
|
||||
t.Fatalf("path = %q, want %q", got, want)
|
||||
}
|
||||
if !strings.Contains(string(outputs[0].Data), `<link rel="stylesheet" href="https://example.com/assets/report.css">`) {
|
||||
t.Fatalf("html = %q, want stylesheet link", outputs[0].Data)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateMarkdownIndexSelectsOnlyMarkdownFile(t *testing.T) {
|
||||
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||
|
||||
|
||||
@@ -1,10 +1,19 @@
|
||||
package markdown
|
||||
|
||||
import "bytes"
|
||||
import (
|
||||
"bytes"
|
||||
"html"
|
||||
)
|
||||
|
||||
func wrapHTML(body []byte) []byte {
|
||||
func wrapHTML(body []byte, cssHref string) []byte {
|
||||
var buf bytes.Buffer
|
||||
buf.WriteString("<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title></title>\n</head>\n<body>\n")
|
||||
buf.WriteString("<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n")
|
||||
if cssHref != "" {
|
||||
buf.WriteString("<link rel=\"stylesheet\" href=\"")
|
||||
buf.WriteString(html.EscapeString(cssHref))
|
||||
buf.WriteString("\">\n")
|
||||
}
|
||||
buf.WriteString("<title></title>\n</head>\n<body>\n")
|
||||
buf.Write(body)
|
||||
buf.WriteString("</body>\n</html>\n")
|
||||
return buf.Bytes()
|
||||
|
||||
@@ -23,8 +23,9 @@ type Request struct {
|
||||
}
|
||||
|
||||
type MarkdownOptions struct {
|
||||
Mode string
|
||||
Input string
|
||||
Mode string
|
||||
Input string
|
||||
CssHref string
|
||||
}
|
||||
|
||||
type Transformer interface {
|
||||
|
||||
Reference in New Issue
Block a user