eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document ordered pipeline operations and retire sequential examples

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-21 22:25:11 +00:00
parent 9184072839
commit 7071102ab7
18 changed files with 431 additions and 526 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
docs/internal/overview.md
View File

@@ -14,11 +14,13 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
output files returned by the runner. Cache and debug collaborators are supplied
at this boundary.
Resolution produces a fixed ordered workflow and a sorted set of artifact
lanes. Preparation constructs the complete module and validator set before the
runner receives source bytes. Source parsing and chunking are serial; extraction
uses a bounded run-wide worker pool, followed by serial per-lane merge and
normalize continuations that may overlap across lanes.
Resolution produces a fixed ordered workflow of steps and globally unique,
sorted artifact lanes. Preparation constructs the complete module and validator
set before the runner receives source bytes. Source parsing and chunking are
serial. Each step then uses a bounded run-wide extraction pool followed by
serial per-lane merge and normalize continuations. A step barrier prevents
later consumers from starting until all earlier lanes are terminal and their
required normalized artifacts have crossed the typed handoff.
## Application Boundary
@@ -42,7 +44,7 @@ normalize continuations that may overlap across lanes.
| Package | Implemented responsibility |
| --- | --- |
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
@@ -111,12 +113,15 @@ this package. Domain-neutral prompt filesystem composition lives in
`internal/framework/promptfs`.
The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
Preparation strictly decodes and identity-validates one normalized JSON
artifact, emits canonical registry JSON to the spell prompt, and records only
its semantic digest and count in prepared metadata. The raw reference remains
independently tracked by pipeline provenance. An absent registry is represented
only by the empty prompt value `{"npcs":[]}`. Spell extraction consumes this
shared registry boundary without changing its public module contract.
External references are strictly decoded and identity-validated during
preparation; generated references are decoded and identity-validated at the
ordered step handoff. Both paths emit canonical registry JSON to operation-time
spell and combat prompt or normalization requests. The framework records
generated identity and bounded producer provenance, while the raw external
reference remains independently tracked by pipeline provenance. An absent
registry is represented only by the empty prompt value `{"npcs":[]}`. Spell
and combat consumers use this shared boundary without changing their public
module contracts.
Generic validators under `internal/modules/generic/validate` provide
unconditional test decisions, JSON syntax validation, and JSON Schema
@@ -144,12 +149,12 @@ Implementation details for all production extensions are in
| Surface | Implemented owners | Internal purpose |
| --- | --- | --- |
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes. |
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. |
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |
Physical layout, cleanup, recovery, and sensitive-data handling are defined
in [Operations](../operations.md). Concrete stage modules receive recorder
in [Operations](../operations.md). Concrete modules receive recorder
interfaces and request data, not physical state roots.
## Focused Documentation