Add Markdown HTML publication
This commit is contained in:
10
docs/cli.md
10
docs/cli.md
@@ -20,7 +20,7 @@ distributor inspect
|
|||||||
|
|
||||||
`version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time.
|
`version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time.
|
||||||
|
|
||||||
`run --config <path>` executes configured local-to-local pipelines that publish source files only.
|
`run --config <path>` executes configured local-to-local pipelines that publish source files, generated HTML files, or both.
|
||||||
|
|
||||||
`run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions without writing files.
|
`run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions without writing files.
|
||||||
|
|
||||||
@@ -28,7 +28,7 @@ distributor inspect
|
|||||||
|
|
||||||
`inspect <path>` validates discovered local source bundles and prints a concise normalized summary.
|
`inspect <path>` validates discovered local source bundles and prints a concise normalized summary.
|
||||||
|
|
||||||
Remote backends and HTML publication are not implemented yet.
|
Remote backends are not implemented yet.
|
||||||
|
|
||||||
## Flag reference
|
## Flag reference
|
||||||
|
|
||||||
@@ -70,3 +70,9 @@ Publish the local example:
|
|||||||
```sh
|
```sh
|
||||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Publish the local HTML example:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/distributor run --config examples/local-html.yml
|
||||||
|
```
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ If `--config` is omitted during run, the built-in default path is:
|
|||||||
/usr/local/etc/distributor/config.yml
|
/usr/local/etc/distributor/config.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
The current implementation supports local-to-local source-file publication. Remote backends and HTML publication are not implemented yet.
|
The current implementation supports local-to-local publication of source files, generated HTML files, or both. Remote backends are not implemented yet.
|
||||||
|
|
||||||
## Minimal config
|
## Minimal config
|
||||||
|
|
||||||
@@ -74,7 +74,7 @@ Backends:
|
|||||||
Destination policy:
|
Destination policy:
|
||||||
|
|
||||||
- `publish.source`: publish source artifacts.
|
- `publish.source`: publish source artifacts.
|
||||||
- `publish.html`: publish generated HTML artifacts.
|
- `publish.html`: publish generated HTML artifacts from Markdown source files.
|
||||||
- `transfer.on_destination_same`: `skip` or `fail`, defaults to `skip`.
|
- `transfer.on_destination_same`: `skip` or `fail`, defaults to `skip`.
|
||||||
- `transfer.on_destination_older`: `replace` or `fail`, defaults to `replace`.
|
- `transfer.on_destination_older`: `replace` or `fail`, defaults to `replace`.
|
||||||
- `transfer.on_destination_newer`: `skip` or `fail`, defaults to `skip`.
|
- `transfer.on_destination_newer`: `skip` or `fail`, defaults to `skip`.
|
||||||
@@ -82,6 +82,8 @@ Destination policy:
|
|||||||
|
|
||||||
When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `transform.markdown_to_html.mode: sidecar` are required.
|
When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `transform.markdown_to_html.mode: sidecar` are required.
|
||||||
|
|
||||||
|
Markdown-to-HTML sidecar generation writes `report.html` for `report.md` and does not mutate the source bundle.
|
||||||
|
|
||||||
## Secrets
|
## Secrets
|
||||||
|
|
||||||
Do not put literal secrets in config files. S3 credentials may refer to environment variable names with:
|
Do not put literal secrets in config files. S3 credentials may refer to environment variable names with:
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
|
|
||||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transfer policy, destination bundle path, and existing destination state.
|
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transfer policy, destination bundle path, and existing destination state.
|
||||||
|
|
||||||
Output is a plan with an action, reason, and selected source outputs. Execution writes selected source files and `.distributor.json` for publish or replacement actions.
|
Output is a plan with an action, reason, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
|
||||||
|
|
||||||
## Actions
|
## Actions
|
||||||
|
|
||||||
@@ -16,7 +16,7 @@ Supported actions are publish new, replace older destination, skip same source,
|
|||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
The current implementation publishes source files only. HTML generation and remote backend execution are not implemented.
|
The current implementation publishes source files and Markdown-to-HTML sidecar outputs. Remote backend execution is not implemented.
|
||||||
|
|
||||||
The package uses `internal/state` for destination comparison and `internal/storage` for IO. It does not parse CLI flags or load config files.
|
The package uses `internal/state` for destination comparison and `internal/storage` for IO. It does not parse CLI flags or load config files.
|
||||||
|
|
||||||
|
|||||||
23
docs/internal/transform.md
Normal file
23
docs/internal/transform.md
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
# Transform
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`internal/transform` defines generated publication artifacts. `internal/transform/markdown` implements Markdown-to-HTML sidecar generation.
|
||||||
|
|
||||||
|
## Inputs and outputs
|
||||||
|
|
||||||
|
Inputs are a validated source bundle and source backend. Outputs include destination path, source path, transform id, generated bytes, SHA-256, and size.
|
||||||
|
|
||||||
|
## Markdown behavior
|
||||||
|
|
||||||
|
Markdown files ending in `.md` generate `.html` files in the same logical directory. Non-Markdown files do not generate outputs. Raw HTML embedded in Markdown is not passed through by the renderer.
|
||||||
|
|
||||||
|
Generated HTML is deterministic for the same source content and transform configuration.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
Transforms do not publish files, mutate source bundles, or write destination state. Publish planning selects and writes transform outputs.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
Before changing transform behavior, inspect tests under `internal/transform`.
|
||||||
@@ -14,6 +14,12 @@ Run the local publication:
|
|||||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Run the local HTML publication:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/distributor run --config examples/local-html.yml
|
||||||
|
```
|
||||||
|
|
||||||
## Filesystem layout
|
## Filesystem layout
|
||||||
|
|
||||||
Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root.
|
Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root.
|
||||||
@@ -22,7 +28,7 @@ The maintained example writes under `workspace/`, which is ignored by Git.
|
|||||||
|
|
||||||
## Destination state
|
## Destination state
|
||||||
|
|
||||||
Each published destination bundle contains `.distributor.json`. This state file records the source manifest and copied source outputs. It is the authoritative marker that a destination path is managed by `distributor`.
|
Each published destination bundle contains `.distributor.json`. This state file records the source manifest, copied source outputs, and generated outputs. It is the authoritative marker that a destination path is managed by `distributor`.
|
||||||
|
|
||||||
`manifest.json` from the source bundle is not copied as destination state.
|
`manifest.json` from the source bundle is not copied as destination state.
|
||||||
|
|
||||||
@@ -36,4 +42,4 @@ If a write fails during local publication, `distributor` removes outputs written
|
|||||||
|
|
||||||
## Caveats
|
## Caveats
|
||||||
|
|
||||||
Only local-to-local source-file publication is implemented. SSH, S3, HTML generation, notification, and force overwrite behavior are not implemented.
|
Only local-to-local publication is implemented. SSH, S3, notification, and force overwrite behavior are not implemented.
|
||||||
|
|||||||
16
examples/local-html.yml
Normal file
16
examples/local-html.yml
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
pipelines:
|
||||||
|
- id: example-html-bundle
|
||||||
|
source:
|
||||||
|
backend: local
|
||||||
|
path: examples/source-bundle
|
||||||
|
destinations:
|
||||||
|
- id: local-html
|
||||||
|
backend: local
|
||||||
|
path: workspace/published/html-bundle
|
||||||
|
publish:
|
||||||
|
source: false
|
||||||
|
html: true
|
||||||
|
transform:
|
||||||
|
markdown_to_html:
|
||||||
|
enabled: true
|
||||||
|
mode: sidecar
|
||||||
5
go.mod
5
go.mod
@@ -2,4 +2,7 @@ module gitea.maximumdirect.net/eric/distributor
|
|||||||
|
|
||||||
go 1.26
|
go 1.26
|
||||||
|
|
||||||
require gopkg.in/yaml.v3 v3.0.1
|
require (
|
||||||
|
github.com/yuin/goldmark v1.8.2
|
||||||
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
|
)
|
||||||
|
|||||||
2
go.sum
2
go.sum
@@ -1,3 +1,5 @@
|
|||||||
|
github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
|
||||||
|
github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/adapters/local"
|
"gitea.maximumdirect.net/eric/distributor/internal/adapters/local"
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||||
@@ -73,6 +74,7 @@ func runConfig(ctx context.Context, cfg config.Config, options RunOptions) error
|
|||||||
DestinationBackend: destinationBackend,
|
DestinationBackend: destinationBackend,
|
||||||
DestinationBundlePath: sourceBundle.RootRelativePath,
|
DestinationBundlePath: sourceBundle.RootRelativePath,
|
||||||
Publish: *destination.Publish,
|
Publish: *destination.Publish,
|
||||||
|
Transform: destination.Transform,
|
||||||
Transfer: destination.Transfer,
|
Transfer: destination.Transfer,
|
||||||
DistributorVersion: Version,
|
DistributorVersion: Version,
|
||||||
}
|
}
|
||||||
@@ -102,5 +104,16 @@ func writePlanLine(w io.Writer, plan publish.Plan, planErr error) {
|
|||||||
fmt.Fprintf(w, " - bundle=%s destination=%s action=error reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, planErr.Error())
|
fmt.Fprintf(w, " - bundle=%s destination=%s action=error reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, planErr.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, " - bundle=%s destination=%s action=%s outputs=%d reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, plan.Action, len(plan.Outputs), plan.Reason)
|
fmt.Fprintf(w, " - bundle=%s destination=%s action=%s outputs=%s reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, plan.Action, outputSummary(plan.Outputs), plan.Reason)
|
||||||
|
}
|
||||||
|
|
||||||
|
func outputSummary(outputs []publish.Output) string {
|
||||||
|
if len(outputs) == 0 {
|
||||||
|
return "none"
|
||||||
|
}
|
||||||
|
paths := make([]string, 0, len(outputs))
|
||||||
|
for _, output := range outputs {
|
||||||
|
paths = append(paths, output.DestinationPath)
|
||||||
|
}
|
||||||
|
return strings.Join(paths, ",")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -34,7 +35,7 @@ func TestRunDryRunPrintsConfigSummary(t *testing.T) {
|
|||||||
for _, want := range []string{
|
for _, want := range []string{
|
||||||
"Configured pipelines: 1",
|
"Configured pipelines: 1",
|
||||||
"- reports: source=local bundles=1 destinations=1",
|
"- reports: source=local bundles=1 destinations=1",
|
||||||
"bundle=. destination=archive action=publish_new",
|
"bundle=. destination=archive action=publish_new outputs=report.md,summary.txt",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(output, want) {
|
if !strings.Contains(output, want) {
|
||||||
t.Fatalf("Run() output = %q, want substring %q", output, want)
|
t.Fatalf("Run() output = %q, want substring %q", output, want)
|
||||||
@@ -72,6 +73,103 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunPublishesHTMLOnly(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||||
|
|
||||||
|
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, false, true)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
assertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<h1>Report</h1>")
|
||||||
|
if _, err := os.Stat(filepath.Join(destinationRoot, "report.md")); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("report.md stat error = %v, want not exist", err)
|
||||||
|
}
|
||||||
|
destinationState := readStateFile(t, filepath.Join(destinationRoot, ".distributor.json"))
|
||||||
|
if got, want := len(destinationState.Outputs), 1; got != want {
|
||||||
|
t.Fatalf("state output count = %d, want %d", got, want)
|
||||||
|
}
|
||||||
|
output := destinationState.Outputs[0]
|
||||||
|
if output.Kind != state.OutputKindGenerated || output.Transform != "markdown_to_html" || output.Path != "report.html" || output.SourcePath != "report.md" {
|
||||||
|
t.Fatalf("generated output metadata = %#v", output)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunPublishesSourceAndHTML(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||||
|
|
||||||
|
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
|
||||||
|
assertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<p>Sunny.</p>")
|
||||||
|
assertFile(t, filepath.Join(destinationRoot, "summary.txt"), "Summary\n")
|
||||||
|
destinationState := readStateFile(t, filepath.Join(destinationRoot, ".distributor.json"))
|
||||||
|
if got, want := len(destinationState.Outputs), 3; got != want {
|
||||||
|
t.Fatalf("state output count = %d, want %d", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunDoesNotMutateSourceBundle(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||||
|
sourcePath := filepath.Join(sourceRoot, "report.md")
|
||||||
|
before, err := os.ReadFile(sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read source before: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
after, err := os.ReadFile(sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read source after: %v", err)
|
||||||
|
}
|
||||||
|
if string(after) != string(before) {
|
||||||
|
t.Fatalf("source changed from %q to %q", before, after)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunFailsOnOutputPathCollision(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
writeSourceBundle(t, sourceRoot, "", testBundleOptions{ExtraFiles: []testFile{{Path: "report.html", Data: "<p>source html</p>\n"}}})
|
||||||
|
|
||||||
|
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "destination output path collision") {
|
||||||
|
t.Fatalf("Run() error = %v, want collision", err)
|
||||||
|
}
|
||||||
|
if entries, err := os.ReadDir(destinationRoot); err != nil || len(entries) != 0 {
|
||||||
|
t.Fatalf("destination entries = %v err=%v, want empty", entries, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunDryRunReportsGeneratedOutputs(t *testing.T) {
|
||||||
|
sourceRoot := t.TempDir()
|
||||||
|
destinationRoot := t.TempDir()
|
||||||
|
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
err := Run(context.Background(), RunOptions{
|
||||||
|
ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, false, true),
|
||||||
|
DryRun: true,
|
||||||
|
Stdout: &stdout,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "outputs=report.html") {
|
||||||
|
t.Fatalf("stdout = %q, want generated output path", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunSkipsWhenDestinationStateMatches(t *testing.T) {
|
func TestRunSkipsWhenDestinationStateMatches(t *testing.T) {
|
||||||
sourceRoot := t.TempDir()
|
sourceRoot := t.TempDir()
|
||||||
destinationRoot := t.TempDir()
|
destinationRoot := t.TempDir()
|
||||||
@@ -194,8 +292,14 @@ func TestRunDryRunDoesNotWrite(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type testBundleOptions struct {
|
type testBundleOptions struct {
|
||||||
ID string
|
ID string
|
||||||
Created time.Time
|
Created time.Time
|
||||||
|
ExtraFiles []testFile
|
||||||
|
}
|
||||||
|
|
||||||
|
type testFile struct {
|
||||||
|
Path string
|
||||||
|
Data string
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptions) bundle.Manifest {
|
func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptions) bundle.Manifest {
|
||||||
@@ -217,6 +321,12 @@ func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptio
|
|||||||
{path: "report.md", data: "# Report\nSunny.\n"},
|
{path: "report.md", data: "# Report\nSunny.\n"},
|
||||||
{path: "summary.txt", data: "Summary\n"},
|
{path: "summary.txt", data: "Summary\n"},
|
||||||
}
|
}
|
||||||
|
for _, extra := range opts.ExtraFiles {
|
||||||
|
files = append(files, struct {
|
||||||
|
path string
|
||||||
|
data string
|
||||||
|
}{path: extra.Path, data: extra.Data})
|
||||||
|
}
|
||||||
manifestFiles := make([]bundle.ManifestFile, 0, len(files))
|
manifestFiles := make([]bundle.ManifestFile, 0, len(files))
|
||||||
for _, file := range files {
|
for _, file := range files {
|
||||||
if err := os.WriteFile(filepath.Join(bundleRoot, filepath.FromSlash(file.path)), []byte(file.data), 0o600); err != nil {
|
if err := os.WriteFile(filepath.Join(bundleRoot, filepath.FromSlash(file.path)), []byte(file.data), 0o600); err != nil {
|
||||||
@@ -248,6 +358,19 @@ func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptio
|
|||||||
|
|
||||||
func writeLocalConfig(t *testing.T, sourceRoot, destinationRoot string) string {
|
func writeLocalConfig(t *testing.T, sourceRoot, destinationRoot string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
return writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeLocalConfigWithPolicy(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool) string {
|
||||||
|
t.Helper()
|
||||||
|
transformConfig := ""
|
||||||
|
if publishHTML {
|
||||||
|
transformConfig = `
|
||||||
|
transform:
|
||||||
|
markdown_to_html:
|
||||||
|
enabled: true
|
||||||
|
mode: sidecar`
|
||||||
|
}
|
||||||
return writeConfigFile(t, `
|
return writeConfigFile(t, `
|
||||||
pipelines:
|
pipelines:
|
||||||
- id: reports
|
- id: reports
|
||||||
@@ -259,8 +382,8 @@ pipelines:
|
|||||||
backend: local
|
backend: local
|
||||||
path: `+destinationRoot+`
|
path: `+destinationRoot+`
|
||||||
publish:
|
publish:
|
||||||
source: true
|
source: `+fmt.Sprintf("%t", publishSource)+`
|
||||||
html: false
|
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
|
||||||
`)
|
`)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -341,3 +464,14 @@ func assertFile(t *testing.T, path, want string) {
|
|||||||
t.Fatalf("%s = %q, want %q", path, got, want)
|
t.Fatalf("%s = %q, want %q", path, got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func assertFileContains(t *testing.T, path, want string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read file %s: %v", path, err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(data), want) {
|
||||||
|
t.Fatalf("%s = %q, want substring %q", path, data, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -265,6 +265,7 @@ func TestExampleConfigsLoad(t *testing.T) {
|
|||||||
for _, path := range []string{
|
for _, path := range []string{
|
||||||
"../../examples/local-to-local.yml",
|
"../../examples/local-to-local.yml",
|
||||||
"../../examples/local-publish.yml",
|
"../../examples/local-publish.yml",
|
||||||
|
"../../examples/local-html.yml",
|
||||||
"../../examples/fan-out.yml",
|
"../../examples/fan-out.yml",
|
||||||
} {
|
} {
|
||||||
t.Run(path, func(t *testing.T) {
|
t.Run(path, func(t *testing.T) {
|
||||||
|
|||||||
@@ -36,20 +36,23 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
|
|||||||
_ = req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, managedOutputPaths(writtenOutputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true})
|
_ = req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, managedOutputPaths(writtenOutputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true})
|
||||||
}
|
}
|
||||||
for _, output := range plan.Outputs {
|
for _, output := range plan.Outputs {
|
||||||
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, output.SourcePath)
|
|
||||||
if err != nil {
|
|
||||||
cleanup()
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
destinationPath, err := storage.Join(req.DestinationBundlePath, output.DestinationPath)
|
destinationPath, err := storage.Join(req.DestinationBundlePath, output.DestinationPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
cleanup()
|
cleanup()
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
data, err := req.SourceBackend.ReadFile(ctx, sourcePath)
|
data := output.Data
|
||||||
if err != nil {
|
if output.Kind == state.OutputKindSource {
|
||||||
cleanup()
|
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, output.SourcePath)
|
||||||
return err
|
if err != nil {
|
||||||
|
cleanup()
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
data, err = req.SourceBackend.ReadFile(ctx, sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
cleanup()
|
||||||
|
return err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if _, err := req.DestinationBackend.WriteFile(ctx, destinationPath, data, storage.WriteOptions{Overwrite: false, PreferAtomic: true}); err != nil {
|
if _, err := req.DestinationBackend.WriteFile(ctx, destinationPath, data, storage.WriteOptions{Overwrite: false, PreferAtomic: true}); err != nil {
|
||||||
cleanup()
|
cleanup()
|
||||||
|
|||||||
@@ -1,29 +1,62 @@
|
|||||||
package publish
|
package publish
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/transform"
|
||||||
)
|
)
|
||||||
|
|
||||||
func PlanSourceOutputs(req Request) ([]Output, error) {
|
func PlanOutputs(ctx context.Context, req Request) ([]Output, error) {
|
||||||
if !req.Publish.Source {
|
var outputs []Output
|
||||||
return nil, nil
|
if req.Publish.Source {
|
||||||
}
|
sourceOutputs, err := PlanSourceOutputs(req)
|
||||||
outputs := make([]Output, 0, len(req.SourceBundle.Manifest.Files))
|
if err != nil {
|
||||||
seen := make(map[string]struct{}, len(req.SourceBundle.Manifest.Files))
|
return nil, err
|
||||||
for _, file := range req.SourceBundle.Manifest.Files {
|
|
||||||
if _, exists := seen[file.Path]; exists {
|
|
||||||
return nil, fmt.Errorf("destination output path collision: %s", file.Path)
|
|
||||||
}
|
}
|
||||||
seen[file.Path] = struct{}{}
|
outputs = append(outputs, sourceOutputs...)
|
||||||
|
}
|
||||||
|
if req.Publish.HTML {
|
||||||
|
generatedOutputs, err := markdownTransformer().Generate(ctx, transform.Request{
|
||||||
|
SourceBundle: req.SourceBundle,
|
||||||
|
SourceBackend: req.SourceBackend,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(generatedOutputs) == 0 {
|
||||||
|
return nil, fmt.Errorf("publish html requested but no markdown source files were found")
|
||||||
|
}
|
||||||
|
for _, generated := range generatedOutputs {
|
||||||
|
outputs = append(outputs, Output{
|
||||||
|
SourcePath: generated.SourcePath,
|
||||||
|
DestinationPath: generated.Path,
|
||||||
|
Kind: state.OutputKindGenerated,
|
||||||
|
Transform: generated.Transform,
|
||||||
|
Data: generated.Data,
|
||||||
|
SHA256: generated.SHA256,
|
||||||
|
Size: generated.Size,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := rejectOutputCollisions(outputs); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return outputs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func PlanSourceOutputs(req Request) ([]Output, error) {
|
||||||
|
outputs := make([]Output, 0, len(req.SourceBundle.Manifest.Files))
|
||||||
|
for _, file := range req.SourceBundle.Manifest.Files {
|
||||||
if err := storage.ValidatePath(file.Path); err != nil {
|
if err := storage.ValidatePath(file.Path); err != nil {
|
||||||
return nil, fmt.Errorf("destination output path %q: %w", file.Path, err)
|
return nil, fmt.Errorf("destination output path %q: %w", file.Path, err)
|
||||||
}
|
}
|
||||||
outputs = append(outputs, Output{
|
outputs = append(outputs, Output{
|
||||||
SourcePath: file.Path,
|
SourcePath: file.Path,
|
||||||
DestinationPath: file.Path,
|
DestinationPath: file.Path,
|
||||||
|
Kind: state.OutputKindSource,
|
||||||
SHA256: file.SHA256,
|
SHA256: file.SHA256,
|
||||||
Size: file.Size,
|
Size: file.Size,
|
||||||
})
|
})
|
||||||
@@ -31,13 +64,28 @@ func PlanSourceOutputs(req Request) ([]Output, error) {
|
|||||||
return outputs, nil
|
return outputs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func rejectOutputCollisions(outputs []Output) error {
|
||||||
|
seen := make(map[string]struct{}, len(outputs))
|
||||||
|
for _, output := range outputs {
|
||||||
|
if err := storage.ValidatePath(output.DestinationPath); err != nil {
|
||||||
|
return fmt.Errorf("destination output path %q: %w", output.DestinationPath, err)
|
||||||
|
}
|
||||||
|
if _, exists := seen[output.DestinationPath]; exists {
|
||||||
|
return fmt.Errorf("destination output path collision: %s", output.DestinationPath)
|
||||||
|
}
|
||||||
|
seen[output.DestinationPath] = struct{}{}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func stateOutputs(outputs []Output) []state.OutputFile {
|
func stateOutputs(outputs []Output) []state.OutputFile {
|
||||||
files := make([]state.OutputFile, 0, len(outputs))
|
files := make([]state.OutputFile, 0, len(outputs))
|
||||||
for _, output := range outputs {
|
for _, output := range outputs {
|
||||||
files = append(files, state.OutputFile{
|
files = append(files, state.OutputFile{
|
||||||
Path: output.DestinationPath,
|
Path: output.DestinationPath,
|
||||||
Kind: state.OutputKindSource,
|
Kind: output.Kind,
|
||||||
SourcePath: output.SourcePath,
|
SourcePath: output.SourcePath,
|
||||||
|
Transform: output.Transform,
|
||||||
SHA256: output.SHA256,
|
SHA256: output.SHA256,
|
||||||
Size: output.Size,
|
Size: output.Size,
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -1,29 +1,101 @@
|
|||||||
package publish
|
package publish
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestPlanSourceOutputsRejectsCollision(t *testing.T) {
|
func TestPlanOutputsRejectsCollision(t *testing.T) {
|
||||||
_, err := PlanSourceOutputs(Request{
|
sourceBackend := fake.New()
|
||||||
|
if _, err := sourceBackend.WriteFile(context.Background(), "report.md", []byte("# Report\n"), storage.WriteOptions{}); err != nil {
|
||||||
|
t.Fatalf("WriteFile report.md error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := sourceBackend.WriteFile(context.Background(), "report.html", []byte("<p>source html</p>\n"), storage.WriteOptions{}); err != nil {
|
||||||
|
t.Fatalf("WriteFile report.html error = %v", err)
|
||||||
|
}
|
||||||
|
reportDigest := bundle.FileDigest([]byte("# Report\n"))
|
||||||
|
htmlDigest := bundle.FileDigest([]byte("<p>source html</p>\n"))
|
||||||
|
files := []bundle.ManifestFile{
|
||||||
|
{Path: "report.md", SHA256: reportDigest, Size: 9},
|
||||||
|
{Path: "report.html", SHA256: htmlDigest, Size: 19},
|
||||||
|
}
|
||||||
|
_, err := PlanOutputs(context.Background(), Request{
|
||||||
|
SourceBackend: sourceBackend,
|
||||||
SourceBundle: bundle.Bundle{
|
SourceBundle: bundle.Bundle{
|
||||||
Manifest: bundle.Manifest{
|
Manifest: bundle.Manifest{
|
||||||
SchemaVersion: 1,
|
SchemaVersion: 1,
|
||||||
ID: "bundle",
|
ID: "bundle",
|
||||||
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
|
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
|
||||||
Files: []bundle.ManifestFile{
|
Digest: bundle.BundleDigest(files),
|
||||||
{Path: "report.md", SHA256: "sha256:1111111111111111111111111111111111111111111111111111111111111111", Size: 1},
|
Files: files,
|
||||||
{Path: "report.md", SHA256: "sha256:2222222222222222222222222222222222222222222222222222222222222222", Size: 1},
|
|
||||||
},
|
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
Publish: config.PublishPolicy{Source: true},
|
Publish: config.PublishPolicy{Source: true, HTML: true},
|
||||||
|
Transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{Enabled: true, Mode: config.TransformModeSidecar}},
|
||||||
})
|
})
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("PlanSourceOutputs() error = nil, want collision")
|
t.Fatal("PlanSourceOutputs() error = nil, want collision")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPlanOutputsRejectsHTMLWithoutMarkdown(t *testing.T) {
|
||||||
|
sourceBackend := fake.New()
|
||||||
|
if _, err := sourceBackend.WriteFile(context.Background(), "summary.txt", []byte("Summary\n"), storage.WriteOptions{}); err != nil {
|
||||||
|
t.Fatalf("WriteFile summary.txt error = %v", err)
|
||||||
|
}
|
||||||
|
files := []bundle.ManifestFile{{Path: "summary.txt", SHA256: bundle.FileDigest([]byte("Summary\n")), Size: 8}}
|
||||||
|
_, err := PlanOutputs(context.Background(), Request{
|
||||||
|
SourceBackend: sourceBackend,
|
||||||
|
SourceBundle: bundle.Bundle{Manifest: bundle.Manifest{
|
||||||
|
SchemaVersion: 1,
|
||||||
|
ID: "bundle",
|
||||||
|
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
|
||||||
|
Digest: bundle.BundleDigest(files),
|
||||||
|
Files: files,
|
||||||
|
}},
|
||||||
|
Publish: config.PublishPolicy{HTML: true},
|
||||||
|
Transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{Enabled: true, Mode: config.TransformModeSidecar}},
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("PlanOutputs() error = nil, want no markdown failure")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildRejectsHTMLWithoutTransform(t *testing.T) {
|
||||||
|
sourceBackend := fake.New()
|
||||||
|
destinationBackend := fake.New()
|
||||||
|
if _, err := sourceBackend.WriteFile(context.Background(), "report.md", []byte("# Report\n"), storage.WriteOptions{}); err != nil {
|
||||||
|
t.Fatalf("WriteFile report.md error = %v", err)
|
||||||
|
}
|
||||||
|
files := []bundle.ManifestFile{{Path: "report.md", SHA256: bundle.FileDigest([]byte("# Report\n")), Size: 9}}
|
||||||
|
_, err := Build(context.Background(), Request{
|
||||||
|
PipelineID: "reports",
|
||||||
|
DestinationID: "archive",
|
||||||
|
SourceBackend: sourceBackend,
|
||||||
|
DestinationBackend: destinationBackend,
|
||||||
|
DestinationBundlePath: "",
|
||||||
|
SourceBundle: bundle.Bundle{Manifest: bundle.Manifest{
|
||||||
|
SchemaVersion: 1,
|
||||||
|
ID: "bundle",
|
||||||
|
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
|
||||||
|
Digest: bundle.BundleDigest(files),
|
||||||
|
Files: files,
|
||||||
|
}},
|
||||||
|
Publish: config.PublishPolicy{HTML: true},
|
||||||
|
Transfer: config.TransferPolicy{
|
||||||
|
OnDestinationSame: config.TransferActionSkip,
|
||||||
|
OnDestinationOlder: config.TransferActionReplace,
|
||||||
|
OnDestinationNewer: config.TransferActionSkip,
|
||||||
|
OnConflict: config.TransferActionFail,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Build() error = nil, want missing transform error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -8,6 +8,8 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
"gitea.maximumdirect.net/eric/distributor/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
"gitea.maximumdirect.net/eric/distributor/internal/state"
|
||||||
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/transform"
|
||||||
|
markdowntransform "gitea.maximumdirect.net/eric/distributor/internal/transform/markdown"
|
||||||
)
|
)
|
||||||
|
|
||||||
type Action string
|
type Action string
|
||||||
@@ -29,6 +31,7 @@ type Request struct {
|
|||||||
DestinationBackend storage.Backend
|
DestinationBackend storage.Backend
|
||||||
DestinationBundlePath string
|
DestinationBundlePath string
|
||||||
Publish config.PublishPolicy
|
Publish config.PublishPolicy
|
||||||
|
Transform config.Transform
|
||||||
Transfer config.TransferPolicy
|
Transfer config.TransferPolicy
|
||||||
DistributorVersion string
|
DistributorVersion string
|
||||||
}
|
}
|
||||||
@@ -48,6 +51,9 @@ type Plan struct {
|
|||||||
type Output struct {
|
type Output struct {
|
||||||
SourcePath string
|
SourcePath string
|
||||||
DestinationPath string
|
DestinationPath string
|
||||||
|
Kind string
|
||||||
|
Transform string
|
||||||
|
Data []byte
|
||||||
SHA256 string
|
SHA256 string
|
||||||
Size int64
|
Size int64
|
||||||
}
|
}
|
||||||
@@ -56,7 +62,7 @@ func Build(ctx context.Context, req Request) (Plan, error) {
|
|||||||
if err := validateRequest(req); err != nil {
|
if err := validateRequest(req); err != nil {
|
||||||
return Plan{}, err
|
return Plan{}, err
|
||||||
}
|
}
|
||||||
outputs, err := PlanSourceOutputs(req)
|
outputs, err := PlanOutputs(ctx, req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Plan{}, err
|
return Plan{}, err
|
||||||
}
|
}
|
||||||
@@ -96,15 +102,21 @@ func validateRequest(req Request) error {
|
|||||||
if req.DestinationBackend == nil {
|
if req.DestinationBackend == nil {
|
||||||
return fmt.Errorf("destination backend is required")
|
return fmt.Errorf("destination backend is required")
|
||||||
}
|
}
|
||||||
if req.Publish.HTML {
|
if !req.Publish.Source && !req.Publish.HTML {
|
||||||
return fmt.Errorf("publish html is not implemented")
|
return fmt.Errorf("publish source or html must be enabled")
|
||||||
}
|
}
|
||||||
if !req.Publish.Source {
|
if req.Publish.HTML {
|
||||||
return fmt.Errorf("publish source must be enabled")
|
if req.Transform.MarkdownToHTML == nil || !req.Transform.MarkdownToHTML.Enabled || req.Transform.MarkdownToHTML.Mode != config.TransformModeSidecar {
|
||||||
|
return fmt.Errorf("publish html requires markdown_to_html transform enabled with sidecar mode")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func markdownTransformer() transform.Transformer {
|
||||||
|
return markdowntransform.New()
|
||||||
|
}
|
||||||
|
|
||||||
func actionForComparison(comparison state.Comparison, transfer config.TransferPolicy) (Action, string) {
|
func actionForComparison(comparison state.Comparison, transfer config.TransferPolicy) (Action, string) {
|
||||||
switch comparison.Outcome {
|
switch comparison.Outcome {
|
||||||
case state.OutcomeDestinationAbsent:
|
case state.OutcomeDestinationAbsent:
|
||||||
|
|||||||
57
internal/transform/markdown/markdown.go
Normal file
57
internal/transform/markdown/markdown.go
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
package markdown
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/yuin/goldmark"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/transform"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Transformer struct {
|
||||||
|
renderer goldmark.Markdown
|
||||||
|
}
|
||||||
|
|
||||||
|
func New() *Transformer {
|
||||||
|
return &Transformer{renderer: goldmark.New()}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Transformer) Generate(ctx context.Context, req transform.Request) ([]transform.Output, error) {
|
||||||
|
if t.renderer == nil {
|
||||||
|
t.renderer = goldmark.New()
|
||||||
|
}
|
||||||
|
var outputs []transform.Output
|
||||||
|
for _, file := range req.SourceBundle.Manifest.Files {
|
||||||
|
if !strings.HasSuffix(file.Path, ".md") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, file.Path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
data, err := req.SourceBackend.ReadFile(ctx, sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read markdown source %q: %w", file.Path, err)
|
||||||
|
}
|
||||||
|
var rendered bytes.Buffer
|
||||||
|
if err := t.renderer.Convert(data, &rendered); err != nil {
|
||||||
|
return nil, fmt.Errorf("render markdown source %q: %w", file.Path, err)
|
||||||
|
}
|
||||||
|
html := wrapHTML(rendered.Bytes())
|
||||||
|
outputPath := strings.TrimSuffix(file.Path, ".md") + ".html"
|
||||||
|
outputs = append(outputs, transform.Output{
|
||||||
|
Path: outputPath,
|
||||||
|
SourcePath: file.Path,
|
||||||
|
Transform: transform.MarkdownToHTML,
|
||||||
|
Data: html,
|
||||||
|
SHA256: bundle.FileDigest(html),
|
||||||
|
Size: int64(len(html)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return outputs, nil
|
||||||
|
}
|
||||||
113
internal/transform/markdown/markdown_test.go
Normal file
113
internal/transform/markdown/markdown_test.go
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
package markdown
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/transform"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestGenerateMarkdownSidecar(t *testing.T) {
|
||||||
|
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||||
|
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
if got, want := len(outputs), 1; got != want {
|
||||||
|
t.Fatalf("output count = %d, want %d", got, want)
|
||||||
|
}
|
||||||
|
output := outputs[0]
|
||||||
|
if output.Path != "report.html" {
|
||||||
|
t.Fatalf("path = %q, want report.html", output.Path)
|
||||||
|
}
|
||||||
|
if output.SourcePath != "report.md" || output.Transform != transform.MarkdownToHTML {
|
||||||
|
t.Fatalf("metadata = %#v", output)
|
||||||
|
}
|
||||||
|
html := string(output.Data)
|
||||||
|
for _, want := range []string{"<!doctype html>", "<h1>Title</h1>", "<p>Hello.</p>"} {
|
||||||
|
if !strings.Contains(html, want) {
|
||||||
|
t.Fatalf("html = %q, want substring %q", html, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if output.SHA256 != bundle.FileDigest(output.Data) || output.Size != int64(len(output.Data)) {
|
||||||
|
t.Fatalf("digest/size metadata = %s/%d", output.SHA256, output.Size)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGenerateIgnoresNonMarkdown(t *testing.T) {
|
||||||
|
backend := fake.New()
|
||||||
|
if _, err := backend.WriteFile(context.Background(), "summary.txt", []byte("Summary\n"), storage.WriteOptions{}); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
sourceBundle := bundle.Bundle{Manifest: bundle.Manifest{
|
||||||
|
SchemaVersion: 1,
|
||||||
|
ID: "bundle",
|
||||||
|
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
|
||||||
|
Files: []bundle.ManifestFile{{Path: "summary.txt", SHA256: bundle.FileDigest([]byte("Summary\n")), Size: 8}},
|
||||||
|
}}
|
||||||
|
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(outputs) != 0 {
|
||||||
|
t.Fatalf("outputs = %#v, want none", outputs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGenerateDoesNotPassRawHTML(t *testing.T) {
|
||||||
|
backend, sourceBundle := markdownFixture(t, "# Title\n\n<script>alert('x')</script>\n")
|
||||||
|
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
html := string(outputs[0].Data)
|
||||||
|
if strings.Contains(html, "<script>") {
|
||||||
|
t.Fatalf("html contains raw script: %q", html)
|
||||||
|
}
|
||||||
|
if !strings.Contains(html, "raw HTML omitted") && !strings.Contains(html, "<script>") {
|
||||||
|
t.Fatalf("html = %q, want raw HTML disabled or escaped", html)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGenerateDeterministicOutput(t *testing.T) {
|
||||||
|
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
|
||||||
|
first, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
second, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
if string(first[0].Data) != string(second[0].Data) {
|
||||||
|
t.Fatalf("outputs differ:\n%s\n%s", first[0].Data, second[0].Data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func markdownFixture(t *testing.T, markdown string) (*fake.Backend, bundle.Bundle) {
|
||||||
|
t.Helper()
|
||||||
|
backend := fake.New()
|
||||||
|
if _, err := backend.WriteFile(context.Background(), "report.md", []byte(markdown), storage.WriteOptions{}); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := backend.WriteFile(context.Background(), "summary.txt", []byte("Summary\n"), storage.WriteOptions{}); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
files := []bundle.ManifestFile{
|
||||||
|
{Path: "report.md", SHA256: bundle.FileDigest([]byte(markdown)), Size: int64(len(markdown))},
|
||||||
|
{Path: "summary.txt", SHA256: bundle.FileDigest([]byte("Summary\n")), Size: 8},
|
||||||
|
}
|
||||||
|
manifest := bundle.Manifest{
|
||||||
|
SchemaVersion: 1,
|
||||||
|
ID: "bundle",
|
||||||
|
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
|
||||||
|
Files: files,
|
||||||
|
}
|
||||||
|
manifest.Digest = bundle.BundleDigest(manifest.Files)
|
||||||
|
return backend, bundle.Bundle{Manifest: manifest}
|
||||||
|
}
|
||||||
11
internal/transform/markdown/template.go
Normal file
11
internal/transform/markdown/template.go
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
package markdown
|
||||||
|
|
||||||
|
import "bytes"
|
||||||
|
|
||||||
|
func wrapHTML(body []byte) []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
buf.WriteString("<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title></title>\n</head>\n<body>\n")
|
||||||
|
buf.Write(body)
|
||||||
|
buf.WriteString("</body>\n</html>\n")
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
3
internal/transform/plan.go
Normal file
3
internal/transform/plan.go
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
package transform
|
||||||
|
|
||||||
|
const MarkdownToHTML = "markdown_to_html"
|
||||||
30
internal/transform/registry.go
Normal file
30
internal/transform/registry.go
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
package transform
|
||||||
|
|
||||||
|
import "fmt"
|
||||||
|
|
||||||
|
type Registry struct {
|
||||||
|
transformers map[string]Transformer
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewRegistry() *Registry {
|
||||||
|
return &Registry{transformers: make(map[string]Transformer)}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) Register(name string, transformer Transformer) error {
|
||||||
|
if name == "" {
|
||||||
|
return fmt.Errorf("transform name is required")
|
||||||
|
}
|
||||||
|
if transformer == nil {
|
||||||
|
return fmt.Errorf("transformer %s is nil", name)
|
||||||
|
}
|
||||||
|
if _, exists := r.transformers[name]; exists {
|
||||||
|
return fmt.Errorf("transform %s is already registered", name)
|
||||||
|
}
|
||||||
|
r.transformers[name] = transformer
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) Get(name string) (Transformer, bool) {
|
||||||
|
transformer, ok := r.transformers[name]
|
||||||
|
return transformer, ok
|
||||||
|
}
|
||||||
26
internal/transform/transform.go
Normal file
26
internal/transform/transform.go
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
package transform
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||||
|
"gitea.maximumdirect.net/eric/distributor/internal/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Output struct {
|
||||||
|
Path string
|
||||||
|
SourcePath string
|
||||||
|
Transform string
|
||||||
|
Data []byte
|
||||||
|
SHA256 string
|
||||||
|
Size int64
|
||||||
|
}
|
||||||
|
|
||||||
|
type Request struct {
|
||||||
|
SourceBundle bundle.Bundle
|
||||||
|
SourceBackend storage.Backend
|
||||||
|
}
|
||||||
|
|
||||||
|
type Transformer interface {
|
||||||
|
Generate(ctx context.Context, req Request) ([]Output, error)
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user