eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document ordered pipeline operations and retire sequential examples

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-21 22:25:11 +00:00
parent 9184072839
commit 7071102ab7
18 changed files with 431 additions and 526 deletions
Download Patch File Download Diff File Expand all files Collapse all files

159
docs/config.md
View File

@@ -22,9 +22,8 @@ The explicit-path option is defined in the [CLI reference](cli.md).
- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
- [D&D NPC configuration](../examples/dnd-npcs.config.yml)
- [Sequential D&D NPC and spell configuration](../examples/dnd-npc-spell-sequential.config.yml)
- [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml)
- [Sequential D&D NPC and combat-turn configuration](../examples/dnd-npc-combat-sequential.config.yml)
- [D&D NPC-grounded spell and combat configuration](../examples/dnd-npc-grounded.config.yml)
All are complete version 3 files. The fragments below illustrate individual
fields and are not alternate complete configurations.
@@ -140,13 +139,64 @@ Pipeline fields:
- `input`: required module binding.
- `chunk`: optional module binding. Default module is `generic`.
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
lane definitions.
- `artifacts`: the artifact lane map for a single-step pipeline. It is treated
as an implicit step with the stable ID `default`.
- `steps`: an ordered, non-empty list of step definitions. A pipeline may use
`steps` or `artifacts`, but not both. Step IDs must be unique after trimming.
- `output`: optional module binding. Default module is `json`.
- `references`: optional map of reference slot names to reference paths. These
bindings are defaults for eligible pipeline targets that declare the matching
slot.
Each explicit step contains an `id`, an optional `references` map, and a
non-empty `artifacts` map. Steps share the pipeline input, chunk plan, worker
budget, output encoder, manifest, and failure boundary. Lanes within a step
retain the fixed extract, validate, merge, validate, normalize, and validate
workflow; the next step starts only after the current step is terminal.
Generated references use the structured `artifact` source form to identify one
accepted normalized lane from an earlier step:
```yaml
pipelines:
dnd-npc-grounded:
input: seriatim
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
```
The generated binding is explicit and typed; it is not inferred from module,
lane, or slot names. It may be declared at step scope, applying to every
selected target in that step that declares the slot, or at one target's
`references` map. A producer may fan out to compatible target slots, but a
slot accepts only one producer. A producer must be in an earlier step, and a
configured generated dependency is required even when the consumer slot is
otherwise optional. Aggregating several producer artifacts is unsupported.
The producer codec supplies the artifact kind, complete schema identity, media
type, canonical content digest, and size used for compatibility and checkpoint
dependency checks. Only one accepted normalized artifact crosses the boundary;
raw extraction results, rejected output, intermediate values, and validator
diagnostics do not. Generated content is supplied in memory and is never
represented by a filesystem path.
Artifact lane fields:
- `extract`: required module binding.
@@ -164,19 +214,23 @@ See [CLI Reference](cli.md) for command syntax.
Reference bindings are validated against reference slots declared by eligible
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
must be bound after config defaults, target-local references, lane-level
compatibility bindings, and command-line reference overrides are applied.
Config-relative paths are resolved relative to the config file; command-line
reference paths are resolved relative to the current working directory. Bound
files must be UTF-8 text. Reference media types are inferred from file
extensions and checked when a module restricts accepted types; unknown
extensions use `application/octet-stream`. See [CLI Reference](cli.md#run) for
command-line selectors and [Operations](operations.md) for recorded provenance
and sensitive-data handling.
compatibility bindings, step-local references, and command-line reference
overrides are applied. Config-relative paths are resolved relative to the
config file; command-line reference paths are resolved relative to the current
working directory. Bound files must be UTF-8 text. Reference media types are
inferred from file extensions and checked when a module restricts accepted
types; unknown extensions use `application/octet-stream`. See [CLI Reference](cli.md#run)
for command-line selectors and [Operations](operations.md) for recorded
provenance and sensitive-data handling.
Pipeline-level `references` are defaults. They are valid when at least one
eligible target in the full configured pipeline declares the slot, including
chunk, extractor, merger, and normalizer targets. During a run, they apply only
to the selected targets that declare the slot:
to the selected targets that declare the slot. For external bindings, a
step-local binding overrides a pipeline-level default and a target-local
binding retains the existing most-specific precedence. Generated and external
bindings may not resolve to the same target slot, and a step-scoped generated
binding may not be duplicated by a target-local generated binding.
```yaml
pipelines:
@@ -226,6 +280,18 @@ Target-local reference fields use the same map shape at:
Each binding is valid only when that target module declares the slot.
Reference source forms are:
- a scalar string, which is an external file path; or
- an object with only `artifact`, containing trimmed `step` and `lane` IDs for
an earlier producer lane.
Pipeline-level references accept only external paths. Generated references are
valid at step scope or on a target-local `references` map. Their producer's
registered codec is authoritative for schema, media type, and canonical
content identity; an external file is not treated as generated merely because
its bytes decode as the same artifact.
## Module Bindings
Every module binding may use shorthand:
@@ -400,15 +466,17 @@ The extractor uses campaign references only as supporting disambiguation
material; spell casts still must be present in the source transcript.
It also declares an optional `npcs` slot for a normalized NPC artifact. The
slot accepts exactly one `application/json` file no larger than 1 MiB. During
extractor preparation Notarius strictly decodes and identity-validates the
artifact, then gives the model canonical JSON for caster-name grounding.
Registry source references may belong to the NPC-producing session and are
provenance only; they are not spell evidence. The bound registry contributes a
semantic digest and NPC count to extractor metadata and checkpoint identity,
while its names, aliases, content, and path do not appear there. When absent,
the prompt receives the exact empty value `{"npcs":[]}` and no registry
provenance or fingerprint is recorded.
slot accepts exactly one `application/json` artifact no larger than 1 MiB. An
external file is decoded and identity-validated during preparation. A
generated binding is validated at the step handoff and is provided to the
operation through the same reference contract. In both cases, the model
receives canonical JSON for caster-name grounding. Registry source references
may belong to the NPC-producing session and are provenance only; they are not
spell evidence. Generated reference identity and bounded producer provenance
are recorded by the framework; NPC names, aliases, content, and paths are not
copied into manifests or checkpoint decisions. When absent, the prompt receives
the exact empty value `{"npcs":[]}` and no registry provenance or fingerprint
is recorded.
The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
When an overlay is used, bind it independently under
@@ -419,29 +487,34 @@ bound.
The `dnd/npcs` extractor declares the same optional campaign slots as the spell
extractor, but it does not declare the `npcs` registry slot. Its normalizer
accepts no references. To pass an NPC result to a later spell run, bind the
normalized payload explicitly at runtime; the maintained sequential example
documents that operator workflow.
accepts no references. The maintained
[NPC-grounded example](../examples/dnd-npc-grounded.config.yml) binds its
accepted normalized output to the later spell and combat targets through an
explicit ordered step.
The `dnd/combat-turns` extractor declares the optional campaign slots and the
structured `npcs` slot. Campaign references guide only the LLM extraction
stage. The deterministic normalizer declares only `npcs`, whose prepared
immutable registry supports the same actor and target canonicalization. Each
`npcs` slot accepts exactly one UTF-8 `application/json` file no larger than 1
MiB. The registry's source ranges remain provenance for the reference and never
become combat evidence. Binding `npcs` to extraction and normalization is
stage-local, so an operator-driven combat run uses two explicit selectors:
stage. The deterministic normalizer declares only `npcs`, whose operation-time
registry supports the same actor and target canonicalization. Each `npcs` slot
accepts exactly one UTF-8 `application/json` artifact no larger than 1 MiB. The
registry's source ranges remain provenance for the reference and never become
combat evidence. An ordered step binding fans the same generated NPC artifact
out to extraction and normalization:
```text
combat.extract.npcs=<npc-run>/lanes/npcs.json
combat.normalize.npcs=<npc-run>/lanes/npcs.json
```yaml
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
```
When bound, the combat extractor and normalizer record only the registry's
semantic digest and count in their metadata and checkpoint fingerprints; names,
aliases, content, and paths are not recorded there. When absent, the combat
prompt receives the exact empty registry value `{"npcs":[]}` and no registry
provenance or fingerprint is recorded.
When bound, the combat extractor and normalizer receive the generated registry
at operation time. Framework provenance and checkpoint dependencies contain its
kind, schema identity, media type, canonical digest, size, and bounded producer
identity; names, aliases, content, and paths are not recorded there. When
absent, the combat prompt receives the exact empty registry value
`{"npcs":[]}` and no registry provenance or fingerprint is recorded.
## State Surfaces
@@ -572,8 +645,11 @@ Configuration validation checks:
Pipeline resolution additionally checks:
- the pipeline ID exists;
- at least one artifact lane is declared and selected;
- at least one artifact lane is declared and selected in each explicit step;
- `artifacts` and `steps` are mutually exclusive, explicit steps are non-empty,
and step IDs are unique after trimming;
- lanes selected through the CLI exist in the resolved pipeline;
- lane IDs are globally unique across ordered steps;
- required module keys are present;
- module keys are registered for the expected slot;
- module capability requirements are satisfied;
@@ -583,4 +659,7 @@ Pipeline resolution additionally checks:
Scriptorium profile IDs;
- bound reference slots are declared by selected chunk, extractor, merger, or
normalizer targets;
- generated references identify one lane in an earlier step, use a declared
compatible artifact kind, and do not conflict with external or target-local
generated bindings;
- required reference slots are bound for selected targets.