eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan the ordered pipeline steps implementation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-21 20:28:04 +00:00
parent f94ab0a6bf
commit f5618d1f0c
4 changed files with 805 additions and 677 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
docs/roadmap/dnd.md
View File

@@ -1,35 +0,0 @@
# D&D Module Harmonization And Prompt Reuse
This roadmap tracked harmonization of the spell, NPC, and combat-turn lanes.
The implemented behavior is now owned by [Module Internals](../internal/modules.md)
and [LLM Runtime Internals](../internal/llm.md); this document records status
and the one remaining external prerequisite rather than duplicating those
current-behavior references.
## Implementation Status
The offline/runtime work is complete:
- shared prompt assets have explicit per-prompt manifests that drive both
mounting and prompt fingerprints;
- shared prompt inputs, deterministic chunk material preparation, and cited
source traversal are centralized;
- prompt ordering and cache-control policy are aligned across the D&D
extraction prompts, while scene chunking retains its distinct prompt shape;
- validator prerequisite, bounded-diagnostic, and checkpoint policies are
aligned without changing compatibility-sensitive identifiers;
- production registration remains explicit and typed, with grouped composition,
central default-chain ownership, and artifact-specific merge behavior;
- private LLM response schemas remain package-owned and separate from durable
codec schemas; and
- focused tests and repository-wide test, vet, build, and diff checks pass.
## Deferred evaluation
An optional live before/after provider comparison remains deferred because it
requires credentials and a maintained human-reviewed transcript or fixture
set. It is not part of the default test suite or merge gate. If those
prerequisites become available, record only aggregate extraction-review
results, prompt token counts, cache-hit/cache-write metrics, and non-secret
prompt hashes. Never commit transcript content, rendered prompts, credentials,
endpoints, or private reference material.

122
docs/roadmap/future.md
View File

@@ -5,121 +5,17 @@ configuration, operations, internal, and integration docs. This roadmap records
future work only. Items are ordered roughly by current value and specificity, future work only. Items are ordered roughly by current value and specificity,
not as committed release dates. not as committed release dates.
## Near-Term: Ordered Pipeline Steps ## Proposed Next Scope: Ordered Pipeline Steps
Allow one configured pipeline to contain multiple ordered execution steps so Add ordered groups of artifact lanes, canonical generated-artifact references,
accepted artifacts from an earlier step can become generated references for and dependency-aware checkpoint reuse without introducing a general DAG. The
later steps in the same run. This is a bounded extension of the fixed pipeline first production workflow runs normalized D&D NPC extraction before spell and
model, not an arbitrary DAG or general workflow language. combat-turn extraction and supplies that NPC artifact to their declared
reference slots.
Input parsing and chunk planning remain pipeline-wide. Each step selects one or The bounded target, compatibility and architecture decisions, exclusions, and
more artifact lanes; every selected lane completes extraction, validation, acceptance criteria are defined in
merge, normalization, and validation before dependent later steps begin. Lanes [Proposed Scope: Ordered Pipeline Steps](ordered-pipeline-steps.md).
within the same step remain independent and may execute concurrently. The
runner exposes only accepted normalized artifacts across a step boundary; raw
extracts, rejected outputs, and intermediate merge results cannot become
downstream references.
Generated-reference bindings must be explicit in resolved configuration. A
binding identifies an earlier producing lane and one declared reference slot on
a later consuming lane. Resolution must reject missing producers, references to
the same or a later step, incompatible artifact kinds or media types, undeclared
consumer slots, cycles, and ambiguous bindings. A configured external reference
and a generated reference cannot bind the same effective target slot; reject
that pipeline or runtime override instead of applying a precedence rule.
Each effective target slot accepts at most one producer. One generated artifact
may fan out to multiple compatible target slots in a later step; aggregation
from multiple producers into one slot is deferred until a concrete use case
defines deterministic semantics.
Step-scoped reference defaults mirror existing pipeline-level reference
defaults. A step binds an earlier artifact once, and the binding automatically
applies to every target in that step that declares the named slot. Target-local
bindings remain available when only one module should consume the artifact; do
not combine a target-local and step-scoped binding for the same effective slot.
A generated source uses a structured, unambiguous form rather than encoding a
producer into a path-like string. The target configuration shape is:
```yaml
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
```
This snippet shows the step and reference portion of a pipeline; pipeline-wide
input, chunk, output, and other references are omitted. Existing scalar
reference values continue to mean external file paths; the structured
`artifact` form means an accepted normalized artifact from the named earlier
step and lane. Do not infer generated bindings from module keys, matching lane
names, or D&D-specific knowledge in the framework.
The handoff should use the producer artifact's canonical codec representation
and retain its artifact kind, schema identity, media type, content digest, and
producer provenance. Generated references provide context or disambiguation,
not source evidence. They use the existing module-facing reference contract
where possible; typed or domain-specific adapters may validate and prepare a
reference without moving domain concepts into the pipeline framework.
Pipeline identity, manifests, checkpoint dependencies, debug records, and
errors must include step identity and generated-reference provenance. A
downstream checkpoint is reusable only when the upstream artifact identity and
content digest match. Resume should reconstruct an accepted upstream artifact
through its registered codec instead of requiring the producing lane to run
again when its checkpoint is reusable.
### Dependency-aware resume and recomputation
Treat generated-reference bindings as checkpoint dependencies. Reusing an
earlier step is safe only when its existing checkpoint and codec identity are
valid. Reusing a dependent step additionally requires an exact match for every
upstream artifact kind, schema identity, media type, and canonical content
digest it consumed. A changed, missing, rejected, corrupt, or incompatible
producer artifact invalidates all transitive dependent checkpoints; the runner
must never combine a newly produced upstream artifact with stale downstream
output.
Support selectively recomputing one configured step and all of its transitive
dependents while retaining reusable independent and predecessor work. The
operator-facing selection mechanism should identify a stable configured step,
not individual internal stage operations. Resolution must reject a selection
that would omit a required predecessor without a reusable accepted artifact.
Manifests, checkpoint events, and diagnostics should distinguish work that was
executed, reused, or invalidated and record a bounded non-secret reason for
dependency-driven invalidation. Completion order must not affect invalidation,
public artifact ordering, or the set of dependent steps selected for rerun.
If a required producer finishes without an accepted normalized artifact, fail
the entire run with a deterministic dependency error. Do not start any
dependent step. Preserve the upstream rejection or empty-result outcome and
step provenance in the failed run manifest so the cause remains auditable.
The first production workflow is D&D NPC grounding:
1. the first step runs the NPC lane through accepted normalized output; and
2. the second step runs spell and combat-turn lanes, binding that NPC artifact
to the spell extractor and to the combat extractor and normalizer through
their existing `npcs` reference slots.
Spell and combat-turn extraction may run concurrently after the NPC handoff is
available. The NPC artifact may disambiguate participant identity, but it does
not prove that a spell cast or combat turn occurred.
## Near-Term D&D Pipeline ## Near-Term D&D Pipeline

985
docs/roadmap/implementation.md
View File

File diff suppressed because it is too large Load Diff

340
docs/roadmap/ordered-pipeline-steps.md Normal file
View File

@@ -0,0 +1,340 @@
# Proposed Scope: Ordered Pipeline Steps
## Status
Proposed as the next implementation scope. This document refines the
near-term priority in [Future Work](future.md) into a bounded feature target. It
defines desired behavior and scope, not a file-by-file implementation plan.
## Recommendation
Implement ordered pipeline steps, generated artifact references, and
dependency-aware checkpoint reuse as one coherent platform capability. Prove
the capability with the D&D workflow in which accepted normalized NPC output
is produced first and then supplied to spell extraction, combat-turn
extraction, and combat-turn normalization.
This should precede item extraction. It establishes the artifact-handoff and
dependency semantics that the planned item lane will also consume, without
combining two substantial features in one implementation effort.
## Intended Outcome
A configured pipeline may contain multiple ordered steps while retaining one
pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run
manifest, and failure boundary. Every artifact lane still follows the fixed
extract, validate, merge, validate, normalize, and validate lifecycle. Steps
add explicit barriers between groups of lanes; they do not create arbitrary
stage graphs.
An accepted normalized artifact from an earlier step may be bound explicitly
to declared reference slots in a later step. The framework remains
domain-neutral, and generated references remain contextual material rather than
source evidence.
## Fixed Product And Architecture Decisions
### Pipeline shape
- Input parsing and chunk planning remain pipeline-wide and execute once.
- A step contains one or more artifact lanes. Step order is configuration order.
- Lanes within a step remain independent and may use the existing bounded
concurrency model.
- A later step cannot begin until every lane in the current step is terminal
and every generated artifact it requires is accepted and available.
- Public artifact and failure ordering is step order followed by deterministic
lane and source-chunk order, never completion order.
- Output encoding occurs once, after every step succeeds.
- This is not an arbitrary DAG, a general workflow language, concurrent
cross-lane reconciliation, or permission for modules to invoke other modules.
### Configuration model
Existing single-step pipelines remain valid. A top-level `artifacts` map is
treated as an implicit step with stable ID `default`. A pipeline may configure
either `artifacts` or `steps`, but not both. Explicit steps must be non-empty
and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique
across the entire pipeline so output paths, selectors, manifests, errors, and
checkpoint scopes remain unambiguous.
The target configuration shape is:
```yaml
pipelines:
dnd-session:
input: seriatim
chunk: generic
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
output: json
```
Existing scalar reference values continue to represent external file paths.
The structured `artifact` form identifies accepted normalized output from one
earlier step and lane. Generated artifact bindings are allowed at step scope or
at an individual module target; they are not inferred from module keys, lane
names, slot names, or domain knowledge.
A step-scoped reference applies automatically to every selected target in that
step that declares the slot. In the example, one `npcs` binding reaches the
spell extractor plus the combat extractor and normalizer. A target-local
binding is used when only one module should consume the artifact.
Pipeline-level external references remain defaults. Step-local external
references override pipeline-level external defaults, and target-local
external references retain their existing precedence. A generated reference
and an external reference may not resolve to the same effective target slot;
configuration or a runtime override that creates that conflict is invalid.
Likewise, a step-scoped and target-local generated binding cannot both target
the same effective slot.
Each effective target slot accepts at most one producer. One producer may fan
out to multiple compatible slots in a later step. Aggregating several producer
artifacts into one slot is outside this scope.
Reference-slot specs gain optional generated-artifact compatibility metadata.
A generated binding is allowed only when the consumer slot declares the
producer's artifact kind; the producer's registered codec supplies the exact
schema identity and media type used for the handoff. The D&D `npcs` consumer
slots declare the normalized NPC-list artifact kind. Existing external-file
slots and bindings retain their current behavior and do not acquire an artifact
kind merely because their bytes happen to decode as one.
### Resolution and preparation
Resolution validates the complete ordered structure before source processing.
It must reject duplicate identities, missing producers, same-step or forward
references, undeclared slots, reference conflicts, and incompatible artifact
kind, schema, media type, or cardinality constraints that are statically
discoverable. Size is checked when canonical producer bytes exist at handoff.
Ordered steps make cycles structurally impossible; resolution must not
introduce a general graph scheduler to rediscover their order.
The resolved pipeline and its digest include step order, step IDs, lane
membership, generated-reference topology, producer identity, consumer targets,
and existing module and validator policy. Cloning, redaction, canonical JSON,
debug summaries, and manifests preserve the same structure without reference
content or secrets.
All modules and validators are still selected, option-validated, and
constructed before source parsing. Generated content cannot be supplied during
construction because it does not exist yet. The framework therefore augments
the existing operation-request `References` at the step boundary. Consumers
that currently assume an NPC registry is construction-only must accept the
generated registry from their operation request without deferring general
module construction until after upstream work.
Only validation that inherently depends on generated bytes may occur at the
handoff. A handoff validation failure is a contextual framework error and fails
the run before any consumer in that step begins.
### Generated artifact handoff
Only accepted normalized output may cross a step boundary. Raw extraction
responses, rejected artifacts, merge intermediates, and validator diagnostics
cannot be bound as references.
The framework serializes the producer through its registered canonical artifact
codec and constructs one immutable reference item containing:
- the declared target slot;
- canonical artifact bytes and media type;
- artifact kind and schema ID, name, version, and schema digest;
- canonical content digest and size; and
- producer pipeline, step, lane, and module provenance.
A generated binding requires exactly one accepted normalized artifact from its
producer lane. No artifact is a missing dependency, while more than one is a
cardinality error; a typed collection such as an NPC list remains one artifact.
Combining several normalized outputs into one reference is aggregation and is
outside this scope.
The existing slot contract remains authoritative for accepted media types,
maximum size, and cardinality. Generated content is cloned at ownership
boundaries and never exposed through a filesystem path. Manifests and debug
summaries record identities and bounded provenance, not artifact content.
Configuring a generated binding makes that dependency required even when the
consumer module declares the underlying slot optional. An accepted artifact
whose domain collection is empty is still a valid artifact and may be handed
off. If the producer has no accepted normalized artifact, the entire run fails
with a deterministic dependency error and no later step begins.
### Checkpoint reuse and selective recomputation
Generated references participate in downstream checkpoint dependencies by
artifact kind, complete schema identity, media type, and canonical content
digest. The pipeline digest protects topology; stage dependency fingerprints
protect the exact upstream artifact consumed. The runner must never combine a
new or changed producer with stale dependent output.
Compatible producer checkpoints may be decoded through the registered codec
and handed to later steps without rerunning the producer. A missing, rejected,
corrupt, incompatible, or changed producer invalidates every transitive
dependent checkpoint. Independent work remains reusable.
Add one operator control, `--recompute-step <step-id>`, with these semantics:
- it requires checkpoint recording and `--resume`;
- the selected step and all transitive dependents execute rather than reuse
their checkpoints;
- valid required predecessors and unrelated work remain reusable;
- the recompute selection affects loader decisions, not the persistent
checkpoint identity of otherwise identical work; and
- the command fails before dependent execution if a required predecessor has
no reusable accepted artifact.
Existing `--only` behavior remains unchanged for implicit single-step
pipelines. Combining `--only` with explicit multi-step pipelines is outside
this scope and should be rejected with actionable guidance rather than given
implicit dependency-expansion semantics.
Checkpoint events, manifests, and diagnostics distinguish executed, reused,
forced-recomputed, and dependency-invalidated work. Invalidation reasons are
bounded, deterministic, and free of reference content, local paths, or secrets.
Old checkpoint state need not be migrated; it must produce a safe, explicit
cold miss rather than an error or unsafe reuse.
### Failure, cancellation, and concurrency
The existing run-wide worker and provider-call limits apply across every step.
Workers may be reused between steps, but concurrency cannot cross a step
barrier. A framework error cancels started work using the existing bounded
drain behavior and prevents later steps and output encoding. Rejections remain
recorded outcomes, but failure to produce a normalized artifact required by a
generated binding escalates to the run-level dependency error described above.
The failed manifest retains completed upstream outcomes, step and lane
provenance, rejections, checkpoint events, and the dependency failure without
embedding generated artifact content.
## D&D Proving Workflow
The production acceptance workflow has two explicit steps:
1. `identify-npcs` runs the NPC lane through normalization and its complete
validator policy.
2. `grounded-events` receives the canonical NPC artifact in its step-scoped
`npcs` reference and runs spell and combat-turn lanes. The binding reaches
spell extraction, combat-turn extraction, and combat-turn normalization.
Spell and combat-turn lanes may execute concurrently after the handoff. NPC
content may ground names and identities but cannot establish a spell cast or
combat event; source units remain the only event evidence.
The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be
replaced or supplemented by one ordered-pipeline example. Existing module keys,
artifact contracts, reference slot names, prompt IDs, and D&D evidence policy
remain unchanged.
## Included Work
- Configuration parsing, validation, cloning, defaults, redaction, and
documentation for explicit steps and structured generated references.
- Domain-neutral resolved step, dependency, producer, and consumer identities.
- Generated-artifact compatibility metadata on reference-slot contracts,
including D&D NPC-list declarations for every `npcs` consumer.
- Step-aware preparation metadata and runner orchestration.
- Canonical codec handoff into existing reference request contracts.
- Required-dependency failure and bounded provenance behavior.
- Dependency-aware checkpoint reuse, invalidation, events, and selective step
recomputation.
- D&D NPC-first production composition for spell and combat-turn consumers.
- Refactoring the affected D&D consumers so generated NPC references are
available at operation time while retaining early static construction.
- Maintained examples and updates to current architecture, configuration, CLI,
operations, internal, integration, and testing documentation when behavior
lands.
- An ADR recording the bounded ordered-step extension to the fixed pipeline
architecture and its explicit rejection of a general DAG.
## Explicitly Out Of Scope
- D&D item extraction or any other new artifact lane.
- Cross-artifact NPC ID fields or artifact-schema migration machinery.
- Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or
module-controlled scheduling.
- Multiple source inputs, per-step input adapters, per-step chunk plans, or
per-step output encoders.
- Aggregating multiple generated artifacts into one reference slot.
- Optional or best-effort generated dependencies; a configured dependency is
required in this scope.
- Prior-run or cross-pipeline generated references.
- `--only` dependency closure for explicit multi-step pipelines.
- Cross-lane reconciliation or domain concepts in the generic framework.
- Live-provider tests or model-quality changes to D&D prompts.
## Acceptance Criteria
The scope is complete when:
- all existing single-step configurations retain their current behavior;
- explicit step order and dependency topology resolve deterministically and
affect pipeline identity;
- invalid producer, consumer, conflict, ordering, type, schema, media, and
cardinality configurations fail before source processing when statically
discoverable, while content-size violations fail at handoff;
- no consumer step begins before all required generated artifacts are accepted,
canonicalized, and validated for its target slots;
- one producer artifact fans out safely to every compatible target selected by
a step-scoped binding;
- missing required producer output fails the complete run before dependent work;
- changing NPC output invalidates spell and combat-turn checkpoints while
leaving compatible independent work reusable;
- selective step recomputation executes exactly the selected dependency closure
and reports why work was executed, reused, or invalidated;
- the D&D ordered workflow supplies NPC content to spell extraction, combat-turn
extraction, and combat-turn normalization without treating it as evidence;
- completion timing cannot change public ordering, failure selection, or
dependency behavior;
- output, manifests, checkpoints, and debug artifacts contain the required
identities and provenance without leaking generated reference content; and
- repository-wide tests, vet, build, maintained-example checks, and
documentation validation pass.
## Testing Strategy
Tests should protect behavior and invariants rather than the implementation's
internal scheduler shape.
- Configuration contract tests own legacy shorthand, explicit step parsing,
source-form discrimination, conflicts, and redaction.
- Resolution tests own ordering, global lane uniqueness, dependency validation,
slot compatibility, fan-out, cloning, canonical JSON, and digest changes.
- Runner tests own step barriers, within-step bounded concurrency, stable
ordering, cancellation, required-producer failure, and immutable handoff.
- Checkpoint tests own producer decoding, exact dependency matching, transitive
invalidation, forced recomputation, cold misses, and bounded decisions.
- One CLI contract test should cover the recompute control and its invalid
combinations.
- One D&D integration test with offline fake LLM responses should prove the
complete NPC-to-spell-and-combat handoff, including combat normalization.
- Maintained configuration examples should be parsed and resolved through the
production catalog.
Do not add scheduler choreography tests, exact goroutine-count assertions,
complete manifest snapshots, exact diagnostic strings, or duplicated tests for
every invalid configuration at every layer. No test may require credentials or
a live model provider.
## Open Questions
None required to define this scope. Exact internal type names and implementation
decomposition are intentionally not feature-policy decisions.