From 916532100dd229e44f6130f8d7147c635a970f83 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 9 Aug 2026 20:57:28 +0000 Subject: [PATCH] Add D&D consumer documentation --- README.md | 2 + docs/consumers/dnd-pipeline.md | 202 +++++++++++++++++ docs/consumers/subprocess.md | 4 + docs/development.md | 2 +- docs/roadmap/dnd-subprocess-documentation.md | 224 +++++++++++++++++++ 5 files changed, 433 insertions(+), 1 deletion(-) create mode 100644 docs/consumers/dnd-pipeline.md create mode 100644 docs/roadmap/dnd-subprocess-documentation.md diff --git a/README.md b/README.md index 88c7246..22383cb 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,8 @@ demonstrates all implemented D&D lanes and the supporting campaign references. artifact formats. - [Subprocess consumer guide](docs/consumers/subprocess.md) — invoke Notarius from an orchestrator and consume a published result. +- [Complete D&D consumer guide](docs/consumers/dnd-pipeline.md) — run the full + D&D pipeline as a subprocess and discover its structured artifacts. - [Internal overview](docs/internal/overview.md) — implemented component map for maintainers. - [Developer guide](docs/development.md) — contributor orientation and diff --git a/docs/consumers/dnd-pipeline.md b/docs/consumers/dnd-pipeline.md new file mode 100644 index 0000000..727c278 --- /dev/null +++ b/docs/consumers/dnd-pipeline.md @@ -0,0 +1,202 @@ +# Consuming The Complete D&D Pipeline + +Use this workflow when an orchestrator runs the maintained complete D&D +pipeline and consumes its structured JSON artifacts. The generic +[subprocess consumer guide](subprocess.md) owns process-level responsibilities; +this guide connects that workflow to the complete D&D configuration, its +Seriatim input, and its artifact inventory. + +The [CLI reference](../cli.md), [configuration reference](../config.md), +[run-result receipt](../integrations/run-result.md), and +[published JSON output contract](../integrations/json-output.md) remain the +canonical definitions of those public interfaces. + +## Prepare And Validate The Deployment + +Start from the maintained +[complete D&D configuration](../../examples/dnd-complete.config.yml). It uses +the `dnd-session` pipeline and demonstrates every implemented D&D lane, ordered +artifact handoffs, campaign references, chunk-map publication, and evidence +context. + +A deployment must provide its own PromptKit profile and campaign reference +files. Use absolute paths for service and subprocess deployments. In +particular, observe these different resolution rules: + +- reference paths in YAML are resolved relative to the Notarius configuration + file; and +- `promptkit.profile_file` is resolved relative to the Notarius process working + directory. + +Do not copy the repository example's relative profile path into a deployment +without also controlling that working directory. The complete path and profile +rules are defined in [Configuration](../config.md). + +Preflight the deployed configuration before processing sessions and whenever +it changes: + +```sh +notarius config validate \ + --config /absolute/path/to/notarius.yml \ + --pipeline dnd-session +``` + +Provide credentials through the environment or the documented configuration +mechanism. Do not put credentials in command arguments, generated +configuration, or logs. + +## Supply The Transcript + +The complete pipeline consumes a Seriatim JSON document. The +[Seriatim input contract](../integrations/seriatim.md) defines its required +metadata, segments, and validation rules. Preserve segment IDs: D&D artifact +citations use those segment IDs as source-unit ranges. + +When the caller maintains several transcript tiers, use the final trimmed JSON +transcript so extraction operates on the same session content presented to +later consumers. For example, Narratio identifies this implemented artifact as +`narratio.transcript.final_trimmed` and normally stores it at +`transcripts/final.trimmed.json`. + +Notarius generates a stable prompt session from the resolved input module and +the exact input bytes. An ordinary orchestrator should not pass `--session-id`. +Use that override only when intentionally changing the routing relationship +between invocations; it is not a credential or output identity. + +## Run Notarius + +Invoke the pipeline with explicit absolute paths and request its +machine-readable receipt: + +```sh +notarius run dnd-session \ + --config /absolute/path/to/notarius.yml \ + --input /absolute/path/to/transcripts/final.trimmed.json \ + --output-dir /absolute/path/to/notarius-output \ + --json +``` + +The caller should: + +- capture stdout and stderr separately; +- propagate cancellation and impose an operator-appropriate timeout; +- wait for process completion before interpreting stdout; and +- retain stderr for diagnosis without copying secrets or transcript content + into other logs. + +Only exit status 0 permits decoding stdout as a receipt. Ignore stdout after a +nonzero exit because a failed receipt write can leave partial bytes. The +[CLI reference](../cli.md#output-streams-and-exit-statuses) defines the complete +stream and exit-status contract. + +## Discover The Published Bundle + +Decode the successful stdout document as a supported run-result schema. For +the current contract, `schema_version` is `notarius.run-result.v1`. Tolerate +unknown fields allowed by that version, but reject an unsupported schema +version. + +Use the receipt's absolute `output_directory` as the exact run-specific bundle +root. Do not scan the output root for its newest directory, guess a run ID, or +construct a bundle path. Resolve `index_file` beneath `output_directory` and +reject an absolute logical path or any result that escapes the bundle root. + +Read `index.json` and locate each requested lane in `output_files` by its exact +`lane_id`. Do not guess a lane filename. Before decoding a payload: + +1. resolve its descriptor's relative `file` beneath the bundle root with the + same confinement check; +2. verify the descriptor's media type and schema identity against the linked + artifact contract; and +3. decode the payload according to that contract. + +The [published JSON output contract](../integrations/json-output.md) defines +the index and bundle layout. Treat all paths obtained from a decoded external +document as untrusted until confined to their documented root. + +## Complete Artifact Inventory + +When every configured lane is accepted, the complete example publishes these +lane artifacts: + +| Lane ID | Purpose | Canonical contract | +| --- | --- | --- | +| `item-registry` | Canonical registry of encountered items and currency. | [Item registry](../integrations/dnd-item-registry-artifacts.md) | +| `npc-registry` | Canonical registry of named NPCs. | [NPC registry](../integrations/dnd-npc-registry-artifacts.md) | +| `location-registry` | Canonical registry of named locations. | [Location registry](../integrations/dnd-location-registry-artifacts.md) | +| `scene-descriptions` | Classification, title, and summary for each scene. | [Scene descriptions](../integrations/dnd-scene-description-artifacts.md) | +| `item-occurrences` | Source-grounded item discovery, acquisition, use, transfer, and loss events. | [Item occurrences](../integrations/dnd-item-occurrence-artifacts.md) | +| `spells` | Source-grounded spell casts and casters. | [Spell casts](../integrations/dnd-spell-artifacts.md) | +| `combat-turns` | Source-grounded combat turn participation. | [Combat turns](../integrations/dnd-combat-turn-artifacts.md) | +| `npc-occurrences` | Source-grounded NPC interaction occurrences. | [NPC occurrences](../integrations/dnd-npc-occurrence-artifacts.md) | +| `location-occurrences` | Source-grounded location occurrences. | [Location occurrences](../integrations/dnd-location-occurrence-artifacts.md) | +| `enemy-events` | Source-grounded enemy combat events. | [Enemy events](../integrations/dnd-enemy-event-artifacts.md) | + +The JSON encoder always publishes these bundle-management files: + +| File | Purpose | +| --- | --- | +| `index.json` | Discovery document for lane and pipeline-wide artifacts. | +| `manifest.json` | Run provenance and result summaries. | +| `rejected.json` | Rejected pipeline outputs. | +| `warnings.json` | Accepted-output and run warnings. | + +The complete configuration also requests two pipeline-wide artifacts: + +- [`chunk-map.json`](../integrations/chunk-map.md), the accepted chunk plan and + chunk metadata; and +- [`evidence-context.json`](../integrations/evidence-context.md), a reading + excerpt containing the union of selected cited source units and the + configured surrounding window. + +Discover both from their top-level `index.json` descriptors rather than +treating them as lanes. Evidence context is convenient reading material, not +authoritative provenance; citations in the normalized lane payloads remain the +evidence contract. + +Every optional or lane file is published only when its corresponding artifact +is available. A successful process does not guarantee that all configured +lanes were accepted. + +## Decide What Counts As Consumer Success + +Exit status 0 means Notarius completed the pipeline and published its result +bundle. The receipt or bundle may still report warnings, rejected outputs, or +missing lane descriptors. A downstream consumer must define its own required +artifact set explicitly. + +A caller that claims to consume the complete D&D workflow should normally +require all ten lane IDs in the table and verify each descriptor's expected +contract. If any required lane is missing, rejected, or incompatible, fail the +caller's extraction step while retaining the Notarius bundle for diagnosis. A +consumer that needs only a subset may define and document a narrower policy. + +Keep the successful receipt with the complete published bundle. Retain +`manifest.json`, `rejected.json`, `warnings.json`, and captured process logs as +required by the caller's provenance, diagnosis, and retention policies. Avoid +selectively copying payload files without also preserving enough index and +manifest information to identify their originating run and contracts. + +The transcript, lane artifacts, evidence context, manifest, debug data, and +logs can all contain private campaign information. Apply the same access, +publication, and retention controls used for the source transcript. + +## Consumer Checklist + +- Validate the deployed Notarius configuration and `dnd-session` pipeline. +- Pass the final trimmed Seriatim JSON transcript with stable segment IDs. +- Use absolute configuration, input, output-root, profile, and reference paths + in service deployments. +- Capture stdout and stderr separately and enforce cancellation and timeout. +- Parse stdout only after exit status 0. +- Accept only supported receipt, index, and artifact schema versions while + tolerating permitted unknown fields. +- Use the receipt's `output_directory`; never guess the run directory. +- Confine `index_file` and every descriptor path to the published bundle root. +- Discover lanes by `lane_id` and verify descriptor compatibility before + decoding payloads. +- Enforce an explicit required-lane policy and inspect rejections and warnings. +- Preserve the receipt and sufficient bundle provenance for every retained + artifact. +- Protect all transcript-derived files and diagnostic streams as sensitive + campaign data. diff --git a/docs/consumers/subprocess.md b/docs/consumers/subprocess.md index a53e523..3e55b86 100644 --- a/docs/consumers/subprocess.md +++ b/docs/consumers/subprocess.md @@ -6,6 +6,10 @@ statuses, while the [run-result receipt](../integrations/run-result.md) and [Published JSON Output contract](../integrations/json-output.md) own the durable result formats. +For the maintained complete D&D workflow, including its transcript input, +configured lane inventory, and downstream acceptance checklist, see +[Consuming The Complete D&D Pipeline](dnd-pipeline.md). + ## Run And Check The Process Optionally preflight a selected configuration and pipeline before work starts: diff --git a/docs/development.md b/docs/development.md index a6def97..20e8ddf 100644 --- a/docs/development.md +++ b/docs/development.md @@ -18,7 +18,7 @@ implemented component map. | Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. | | Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. | | CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. | -| Building a subprocess caller or changing its result protocol | [Subprocess Consumer Guide](consumers/subprocess.md), [Run Result Receipt](integrations/run-result.md), and [CLI Internals](internal/cli.md) | These separate caller workflow, durable receipt contract, and CLI implementation behavior. | +| Building a subprocess caller or changing its result protocol | [Subprocess Consumer Guide](consumers/subprocess.md), [Complete D&D Consumer Guide](consumers/dnd-pipeline.md), [Run Result Receipt](integrations/run-result.md), and [CLI Internals](internal/cli.md) | These separate generic caller workflow, the complete D&D workflow, the durable receipt contract, and CLI implementation behavior. | | Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. | | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. | | Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. | diff --git a/docs/roadmap/dnd-subprocess-documentation.md b/docs/roadmap/dnd-subprocess-documentation.md new file mode 100644 index 0000000..46abf99 --- /dev/null +++ b/docs/roadmap/dnd-subprocess-documentation.md @@ -0,0 +1,224 @@ +# D&D Subprocess Consumer Documentation + +## Status + +Completed. The target guide is `docs/consumers/dnd-pipeline.md`. + +## Purpose + +Provide one task-oriented guide for applications that run Notarius as a +subprocess to execute the maintained complete D&D pipeline and consume its +published artifacts. The initial concrete consumer is Narratio, but the guide +must describe the public Notarius workflow rather than depend on Narratio +internals. + +The guide should make the safe integration path obvious without duplicating +the CLI, input, receipt, output-bundle, or individual artifact contracts that +already have canonical documentation. + +## Current State + +The public integration surface is documented accurately but is distributed +across several documents: + +- `docs/consumers/subprocess.md` defines the generic subprocess workflow; +- `docs/cli.md` owns commands, flags, stream behavior, and exit statuses; +- `docs/integrations/seriatim.md` owns the accepted transcript input shape; +- `docs/integrations/run-result.md` owns the machine-readable successful-run + receipt; +- `docs/integrations/json-output.md` owns bundle discovery and logical files; +- the D&D integration documents own the individual lane payload contracts; +- `examples/dnd-complete.config.yml` is the maintained complete pipeline. + +A consumer can reconstruct the full workflow from those documents, but there +is no D&D-focused guide that connects the maintained example to its input, +invocation, complete artifact inventory, discovery procedure, and downstream +acceptance decisions. + +## Target Documentation Set + +### Create `docs/consumers/dnd-pipeline.md` + +This document should own the end-to-end consumer workflow for the maintained +complete D&D configuration. It should be useful to Narratio and to another +subprocess orchestrator with the same needs. + +The guide should contain the following sections. + +#### Prerequisites And Deployment Configuration + +- Link to `examples/dnd-complete.config.yml` rather than embedding a second + complete configuration. +- Explain that a deployment must provide the configured PromptKit profile and + campaign reference files. +- Recommend absolute paths for a service or orchestrator deployment. +- Call out the path-resolution distinction explicitly: YAML reference paths + are relative to the Notarius configuration file, while + `promptkit.profile_file` is relative to the Notarius process working + directory. +- Recommend validating the selected configuration and `dnd-session` pipeline + before processing sessions. + +#### Transcript Input + +- State that the complete pipeline consumes a Seriatim JSON document. +- Link to the canonical Seriatim contract for required fields and validation. +- Recommend the caller's final trimmed transcript when the caller maintains + transcript tiers. For Narratio, identify the implemented source as + `narratio.transcript.final_trimmed`, normally stored at + `transcripts/final.trimmed.json`. +- Explain that segment IDs must remain stable because D&D source references + cite those units. +- Explain that Notarius derives its default prompt session from the input + module and exact input bytes and that ordinary callers should not supply + `--session-id`. + +#### Subprocess Invocation + +- Show one concise invocation using `notarius run dnd-session`, explicit + absolute `--config`, `--input`, and `--output-dir` paths, and `--json`. +- Direct callers to capture stdout and stderr separately, propagate + cancellation, impose an operator-appropriate timeout, and wait for process + completion before parsing stdout. +- State that only exit status zero permits receipt decoding and link to the CLI + contract for the complete exit-status definition. +- Recommend retaining stderr and the invocation context for diagnosis without + logging secrets or transcript content. + +#### Receipt And Bundle Discovery + +- Require callers to accept only supported run-result schema versions while + tolerating unknown fields allowed by that version. +- Direct callers to obtain the exact run-specific bundle from the receipt's + absolute `output_directory`; they must not scan for the newest run directory + or construct a run ID. +- Require a confinement check when resolving `index_file` beneath the reported + bundle root. +- Direct callers to discover lane payloads by `lane_id` in `index.json`, then + verify descriptor media type and schema identity before decoding them. +- Explain that descriptor paths are untrusted relative paths and require the + same confinement discipline. + +#### Complete D&D Artifact Inventory + +Include a compact table for the ten lane IDs selected by the maintained +complete configuration: + +- `item-registry`; +- `npc-registry`; +- `location-registry`; +- `scene-descriptions`; +- `item-occurrences`; +- `spells`; +- `combat-turns`; +- `npc-occurrences`; +- `location-occurrences`; +- `enemy-events`. + +For each row, give a one-line purpose and link to the corresponding canonical +D&D artifact contract. Do not copy its fields or schema rules into the +consumer guide. + +Document the four always-published bundle files—`index.json`, `manifest.json`, +`rejected.json`, and `warnings.json`—and the complete example's configured +`chunk-map.json` and `evidence-context.json` pipeline-wide artifacts. Link to +their canonical contracts and distinguish pipeline-wide artifacts from lane +outputs. + +The inventory must say that a file is available only when its corresponding +artifact was accepted and published. It must not imply that process success +guarantees every configured lane. + +#### Downstream Acceptance And Retention + +- Explain that exit status zero can coexist with rejected outputs, warnings, + or absent lane descriptors. +- Require the consumer to define its required lane set explicitly. Recommend + treating all ten lanes as required when the caller claims to consume the + complete D&D workflow, while allowing another consumer to adopt a narrower + documented policy. +- Recommend retaining the receipt, the complete published bundle, and captured + diagnostic streams long enough to support provenance and failure analysis. +- Explain that `evidence-context.json` is a reading excerpt; authoritative + citations remain in lane payloads. +- Treat transcripts, lane artifacts, evidence context, manifests, and logs as + sensitive campaign data. + +#### Compatibility Checklist + +End with a concise checklist covering process exit, receipt schema, path +confinement, pipeline identity, index decoding, required descriptors, +descriptor schema/media compatibility, warnings and rejections, checksums or +retention, and secure handling. Compatibility should be based on published +receipt and artifact contracts rather than parsing a human version string. + +### Update Existing Navigation + +- Add a short link from `docs/consumers/subprocess.md` to the D&D-specific + workflow. Keep generic subprocess policy in the existing document. +- Add the guide to the documentation links in `README.md`. +- Extend the subprocess-consumer row in `docs/development.md` so maintainers + working on the D&D workflow are routed to the new guide and the canonical + contracts. + +### Verify Canonical Contract Documents + +Review the linked integration documents and the complete example while writing +the guide. Correct an integration document only if repository inspection finds +an actual stale contract. Do not move schema definitions, field tables, CLI +flags, or configuration semantics into the new guide. + +## Narratio Alignment + +The guide may name Narratio as the motivating consumer and identify its current +final-trimmed transcript source. It must not claim that Narratio already has a +Notarius adapter or extraction stage. Until that feature is implemented, +Narratio-specific architecture, configuration, stage behavior, manifest +records, and artifact source IDs belong in Narratio's roadmap. + +Once Narratio implements the integration, its own integration documentation +should link to this guide and the durable Notarius contracts instead of +repeating them. + +## Validation + +Documentation implementation should include: + +```sh +go run ./cmd/notarius config validate \ + --config examples/dnd-complete.config.yml \ + --pipeline dnd-session +go test ./... +``` + +Also verify all new and changed relative Markdown links, compare the artifact +inventory directly with the maintained complete configuration, and confirm +that commands and path semantics match the CLI and configuration references. +If the repository still has no automated link checker, record that fact and +perform a focused manual link review. + +## Acceptance Criteria + +- A subprocess integrator can follow one D&D-focused guide from a Seriatim + transcript through safe discovery of every artifact configured by the + complete example. +- The guide makes stdout, stderr, exit-status, receipt, and path-confinement + responsibilities unambiguous. +- The ten configured D&D lanes and both configured pipeline-wide artifacts are + listed and linked to their canonical contracts. +- The guide distinguishes process success from the caller's required-artifact + policy. +- The profile-path and reference-path resolution rules are clearly stated. +- Existing navigation makes the guide discoverable. +- No volatile contract is defined in two places, and no unimplemented Narratio + behavior is presented as current. + +## Non-Goals + +- Implementing or documenting Narratio's future adapter or stage as current + Notarius behavior. +- Adding a new Notarius command, receipt version, output format, or artifact + schema. +- Duplicating the complete configuration or individual D&D payload schemas in + prose. +- Defining a universal partial-result policy for every Notarius consumer.