diff --git a/docs/roadmap/dnd.md b/docs/roadmap/dnd.md deleted file mode 100644 index b3510f7..0000000 --- a/docs/roadmap/dnd.md +++ /dev/null @@ -1,35 +0,0 @@ -# D&D Module Harmonization And Prompt Reuse - -This roadmap tracked harmonization of the spell, NPC, and combat-turn lanes. -The implemented behavior is now owned by [Module Internals](../internal/modules.md) -and [LLM Runtime Internals](../internal/llm.md); this document records status -and the one remaining external prerequisite rather than duplicating those -current-behavior references. - -## Implementation Status - -The offline/runtime work is complete: - -- shared prompt assets have explicit per-prompt manifests that drive both - mounting and prompt fingerprints; -- shared prompt inputs, deterministic chunk material preparation, and cited - source traversal are centralized; -- prompt ordering and cache-control policy are aligned across the D&D - extraction prompts, while scene chunking retains its distinct prompt shape; -- validator prerequisite, bounded-diagnostic, and checkpoint policies are - aligned without changing compatibility-sensitive identifiers; -- production registration remains explicit and typed, with grouped composition, - central default-chain ownership, and artifact-specific merge behavior; -- private LLM response schemas remain package-owned and separate from durable - codec schemas; and -- focused tests and repository-wide test, vet, build, and diff checks pass. - -## Deferred evaluation - -An optional live before/after provider comparison remains deferred because it -requires credentials and a maintained human-reviewed transcript or fixture -set. It is not part of the default test suite or merge gate. If those -prerequisites become available, record only aggregate extraction-review -results, prompt token counts, cache-hit/cache-write metrics, and non-secret -prompt hashes. Never commit transcript content, rendered prompts, credentials, -endpoints, or private reference material. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 431f5a5..adad441 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -5,121 +5,17 @@ configuration, operations, internal, and integration docs. This roadmap records future work only. Items are ordered roughly by current value and specificity, not as committed release dates. -## Near-Term: Ordered Pipeline Steps +## Proposed Next Scope: Ordered Pipeline Steps -Allow one configured pipeline to contain multiple ordered execution steps so -accepted artifacts from an earlier step can become generated references for -later steps in the same run. This is a bounded extension of the fixed pipeline -model, not an arbitrary DAG or general workflow language. +Add ordered groups of artifact lanes, canonical generated-artifact references, +and dependency-aware checkpoint reuse without introducing a general DAG. The +first production workflow runs normalized D&D NPC extraction before spell and +combat-turn extraction and supplies that NPC artifact to their declared +reference slots. -Input parsing and chunk planning remain pipeline-wide. Each step selects one or -more artifact lanes; every selected lane completes extraction, validation, -merge, normalization, and validation before dependent later steps begin. Lanes -within the same step remain independent and may execute concurrently. The -runner exposes only accepted normalized artifacts across a step boundary; raw -extracts, rejected outputs, and intermediate merge results cannot become -downstream references. - -Generated-reference bindings must be explicit in resolved configuration. A -binding identifies an earlier producing lane and one declared reference slot on -a later consuming lane. Resolution must reject missing producers, references to -the same or a later step, incompatible artifact kinds or media types, undeclared -consumer slots, cycles, and ambiguous bindings. A configured external reference -and a generated reference cannot bind the same effective target slot; reject -that pipeline or runtime override instead of applying a precedence rule. -Each effective target slot accepts at most one producer. One generated artifact -may fan out to multiple compatible target slots in a later step; aggregation -from multiple producers into one slot is deferred until a concrete use case -defines deterministic semantics. - -Step-scoped reference defaults mirror existing pipeline-level reference -defaults. A step binds an earlier artifact once, and the binding automatically -applies to every target in that step that declares the named slot. Target-local -bindings remain available when only one module should consume the artifact; do -not combine a target-local and step-scoped binding for the same effective slot. -A generated source uses a structured, unambiguous form rather than encoding a -producer into a path-like string. The target configuration shape is: - -```yaml -steps: - - id: identify-npcs - artifacts: - npcs: - extract: dnd/npcs - normalize: dnd/npcs - - id: grounded-events - references: - npcs: - artifact: - step: identify-npcs - lane: npcs - artifacts: - spells: - extract: dnd/spells - normalize: dnd/spells - combat: - extract: dnd/combat-turns - normalize: dnd/combat-turns -``` - -This snippet shows the step and reference portion of a pipeline; pipeline-wide -input, chunk, output, and other references are omitted. Existing scalar -reference values continue to mean external file paths; the structured -`artifact` form means an accepted normalized artifact from the named earlier -step and lane. Do not infer generated bindings from module keys, matching lane -names, or D&D-specific knowledge in the framework. - -The handoff should use the producer artifact's canonical codec representation -and retain its artifact kind, schema identity, media type, content digest, and -producer provenance. Generated references provide context or disambiguation, -not source evidence. They use the existing module-facing reference contract -where possible; typed or domain-specific adapters may validate and prepare a -reference without moving domain concepts into the pipeline framework. - -Pipeline identity, manifests, checkpoint dependencies, debug records, and -errors must include step identity and generated-reference provenance. A -downstream checkpoint is reusable only when the upstream artifact identity and -content digest match. Resume should reconstruct an accepted upstream artifact -through its registered codec instead of requiring the producing lane to run -again when its checkpoint is reusable. - -### Dependency-aware resume and recomputation - -Treat generated-reference bindings as checkpoint dependencies. Reusing an -earlier step is safe only when its existing checkpoint and codec identity are -valid. Reusing a dependent step additionally requires an exact match for every -upstream artifact kind, schema identity, media type, and canonical content -digest it consumed. A changed, missing, rejected, corrupt, or incompatible -producer artifact invalidates all transitive dependent checkpoints; the runner -must never combine a newly produced upstream artifact with stale downstream -output. - -Support selectively recomputing one configured step and all of its transitive -dependents while retaining reusable independent and predecessor work. The -operator-facing selection mechanism should identify a stable configured step, -not individual internal stage operations. Resolution must reject a selection -that would omit a required predecessor without a reusable accepted artifact. - -Manifests, checkpoint events, and diagnostics should distinguish work that was -executed, reused, or invalidated and record a bounded non-secret reason for -dependency-driven invalidation. Completion order must not affect invalidation, -public artifact ordering, or the set of dependent steps selected for rerun. - -If a required producer finishes without an accepted normalized artifact, fail -the entire run with a deterministic dependency error. Do not start any -dependent step. Preserve the upstream rejection or empty-result outcome and -step provenance in the failed run manifest so the cause remains auditable. - -The first production workflow is D&D NPC grounding: - -1. the first step runs the NPC lane through accepted normalized output; and -2. the second step runs spell and combat-turn lanes, binding that NPC artifact - to the spell extractor and to the combat extractor and normalizer through - their existing `npcs` reference slots. - -Spell and combat-turn extraction may run concurrently after the NPC handoff is -available. The NPC artifact may disambiguate participant identity, but it does -not prove that a spell cast or combat turn occurred. +The bounded target, compatibility and architecture decisions, exclusions, and +acceptance criteria are defined in +[Proposed Scope: Ordered Pipeline Steps](ordered-pipeline-steps.md). ## Near-Term D&D Pipeline diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 8d245c4..e2f1c9c 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,573 +1,500 @@ -# D&D Module Harmonization Implementation Plan +# Implementation Plan: Ordered Pipeline Steps -This document is the executable implementation plan for the target state in -[D&D Module Harmonization And Prompt Reuse](dnd.md). It is written for a coding -agent and must be followed in stage order. Each stage must leave the repository -building and its focused tests passing before the next stage begins. +## Purpose -Current behavior is documented in the -[module internals](../internal/modules.md) and -[LLM runtime internals](../internal/llm.md). The policies in -[Architecture](../policy/architecture.md), -[Testing](../policy/testing.md), and -[Documentation](../policy/documentation.md) govern all stages. +Implement the target state in [Ordered Pipeline Steps](ordered-pipeline-steps.md). +That feature roadmap is the authority for product intent, policy choices, +acceptance criteria, and exclusions. This document turns it into an ordered, +decision-complete implementation sequence for a coding agent. -## Fixed Decisions And Constraints +This is a platform refactor followed by one D&D proving workflow. It does not +include item extraction, a general DAG scheduler, schema migrations for D&D +artifacts, or any other work excluded by the feature roadmap. -The following decisions are complete and are not implementation-time choices: +## Instructions For Every Stage -- Keep spell, NPC, and combat-turn response DTOs, schemas, canonicalization, - codecs, normalizers, and domain validators in their current domain packages. -- Put D&D-only reuse in `internal/modules/dnd/shared`; do not move D&D concepts - into `internal/framework`. -- Keep private LLM response schemas separate from durable codec schemas. Do not - introduce shared JSON Schema fragments or a schema-generation step in this - work. -- Preserve all public module keys, artifact kinds, prompt IDs, schema IDs, - schema versions, media types, reference slot names, durable JSON fields, and - existing reason-code strings. In particular, retain the spell validator's - existing `invalid_source_refs` reason code as a compatibility exception. -- Preserve central ownership and ordering of default validator chains in the - D&D registrar. -- Do not create a generic extractor framework, use reflection for lane - registration, or erase typed artifact relationships outside existing - framework boundaries. -- Exact cache identity means equal ordered message roles, content bytes, and - cache-control values after Scriptorium has rendered the prompt. Enforce this - primarily through one canonical shared asset per shared message and document - the expected order and cache-control policy. -- Scriptorium and provider-specific types remain confined to the LLM runtime, - prompt-asset wiring, and their tests. Production extractors continue to use - only Notarius contracts. -- Default tests remain offline, deterministic, and credential-free. Live model - evaluation is an explicit manual acceptance activity, not part of - `go test ./...`. -- Do not add message-count, common-prefix-length, or complete rendered-prompt - snapshots. They are change detectors rather than durable behavioral tests. -- Update current-behavior documentation only in the stage that changes that - behavior. Do not describe a partially implemented later stage as complete. +Before changing code, read `docs/development.md` and all documents under +`docs/policy/`. Preserve the fixed pipeline lifecycle and the two-zone data +model described there. Use the repository's code knowledge graph for code +discovery before falling back to text search. -## Target Prompt Layout +Implement the stages in order. Each stage must leave the repository formatted, +building, and passing the focused tests it changes. Do not maintain two +competing internal representations merely to reduce a refactor: legacy +top-level `artifacts` is a configuration compatibility form, while ordered +steps are the single resolved, prepared, and runtime representation. -Stage 3 must produce the following ordered messages. All listed shared entries -must refer to one shared embedded file rather than package-local copies. +Follow these cross-stage rules: -| Index | Spell | NPC | Combat turn | Role | Cache control | -| --- | --- | --- | --- | --- | --- | -| 0 | common system | common system | common system | system | none | -| 1 | common extraction evidence | common extraction evidence | common extraction evidence | user | none | -| 2 | common in-world identity | common in-world identity | common in-world identity | user | `ephemeral` | -| 3 | common transcript | common transcript | common transcript | user | `ephemeral` | -| 4 | common campaign references | common campaign references | common campaign references | user | `ephemeral` | -| 5 | common immediate resolution | NPC task | common immediate resolution | user | none | -| 6 | common NPC registry | NPC instructions | common NPC registry | user | `ephemeral` for spell/combat | -| 7 | spell catalog | — | combat task | user | none | -| 8 | spell task | — | combat instructions | user | none | -| 9 | spell instructions | — | — | user | none | +- Keep input parsing, chunk planning, output encoding, run identity, worker + budgets, provider scheduling, and the failure boundary pipeline-wide. +- Preserve the lane lifecycle and existing typed module interfaces. Steps + schedule lanes; they do not permit modules to call other modules. +- Construct every selected module and validator before source parsing. Supply + generated bytes only through cloned operation-request `References`. +- Resolve and report deterministic order as step order, then lane ID, then + source/chunk order. Never expose completion order. +- Treat generated references as context, not source evidence. Never put their + content in manifests, checkpoint decisions, logs, or debug summaries. +- Add narrow, behavior-oriented, offline tests at the stable owner of each new + invariant. Do not add scheduler choreography, exact goroutine counts, full + manifest snapshots, exact error-string snapshots, or live-provider tests. +- Keep configuration file version 3. The new fields are a backward-compatible + extension; checkpoint wire state receives its own explicit compatibility + change in Stage 4. +- Update current-behavior documentation only in Stage 6, when the complete + behavior exists. The ADR may be added in Stage 1 because it records the + architecture decision rather than claiming released behavior. -The common extraction-evidence asset must state, in artifact-neutral language, -that: +## Canonical Internal Model -- transcript units are the only event evidence; -- campaign and registry references may disambiguate but are not evidence; -- every reported factual claim is supported by cited transcript units; -- references use integer `start_unit_id` and `end_unit_id` values; -- `source_id` is omitted because Notarius assigns the current source identity; -- non-contiguous evidence uses multiple narrow ranges rather than a broad - bridge over unrelated conversation; and -- output contains only the configured JSON object and schema-defined fields. +Use the following model throughout the implementation. Exact unexported helper +names may follow local conventions, but their responsibilities and invariants +are fixed here. -The common in-world-identity asset must require the most specific supported -in-world character or creature identity instead of a human player, transcript -speaker, or the GM as an out-of-world person. It may permit campaign references -to disambiguate an identity, but must not establish participation from a -reference alone. NPC-only exclusion rules and relationship rules remain in the -NPC task or instructions. +### Configuration and profile types -The common immediate-resolution asset, used only by spells and combat turns, -must limit an artifact to a declaration/action and its immediate observed -resolution. It must exclude consequences on later turns or elsewhere in the -scene. Spell-only persistent-effect language and combat-only classification -rules remain local. +- Add `PipelineStepProfile` with `ID`, `Artifacts`, and `References` fields. +- Add `Steps []PipelineStepProfile` to `PipelineProfile`; retain top-level + `Artifacts` only as the legacy input form. +- Replace reference-map values in profile and file-config types with a + discriminated `ReferenceSource`. It has exactly one of: + - an external path, represented by the existing scalar YAML form; or + - `ArtifactReference{Step, Lane}`, represented by + `artifact: {step: ..., lane: ...}`. +- Keep CLI `--reference` values external-path overrides. They do not create or + replace generated bindings. `--reference-unbind` may remove an effective + external binding, but must not silently remove a generated dependency. +- Step-level `References` use the same source type as pipeline and target-local + references. Pipeline-level generated references are invalid because there is + no well-defined consumer step and they could imply a forward dependency. -Remove equivalent prose from package-local task and instruction files after it -has moved to a shared asset. Do not retain paraphrased copies. Read each final -prompt as a whole to remove contradictions and preserve all artifact-specific -requirements. +### Resolved and prepared types -The cache-control choices above create three all-lane breakpoints and one -spell/combat breakpoint. Use no more than these four markers so the request -remains portable across the configured backends; do not add module-specific -markers in this work. +- Make `ResolvedPipeline.Steps []ResolvedPipelineStep` the sole lane container. + Each resolved step has a stable `ID`, ordered `ArtifactLanes`, and expanded + generated consumer bindings. Remove the flat `ArtifactLanes` field after all + callers migrate; provide read-only iteration/lookup helpers where callers + need all lanes. +- Resolve legacy top-level `artifacts` as one step named `default`. Do not add a + synthetic step back into effective user configuration output. +- Keep lane IDs globally unique. A producer and consumer are identified by + pipeline ID, step ID, and lane ID; module targets additionally include stage, + module key, and slot name. +- Represent external and generated bindings as distinct resolved variants. + Never encode a producer selector into a path string. +- Mirror resolved steps in `PreparedPipeline`. A prepared step contains its + already-constructed lanes and static external reference sets. Generated + content is held only in run-local state and overlaid onto request references + before a consumer step begins. -## Stage 1: Protect Shared Input Behavior And Baseline Prompt Wiring +### Generated compatibility and handoff -### Goal +- Extend `contracts.ReferenceSlot` with an optional cloned list of accepted + generated `ArtifactKind` values. An empty list means external references only. + Do not infer compatibility by decoding external bytes. +- A generated producer supplies exactly one accepted normalized artifact per + binding in this feature. Zero artifacts is a missing dependency; more than + one is a cardinality error. A collection such as an NPC list is one typed + artifact, not multiple reference items. Supporting aggregation or several + generated items in one slot remains out of scope. +- Use the producer kind's registered `ArtifactCodecSpec` as the canonical schema + identity and media contract. Kind compatibility plus the single registered + codec makes schema compatibility exact; also validate the consumer's media, + size, and cardinality constraints. +- Extend `contracts.ReferenceItem` with optional generated-artifact identity and + producer provenance rather than overloading file origin fields. Include kind, + schema ID/name/version/digest, producer pipeline/step/lane/module, content + digest, media type, and size. Clone byte slices and nested metadata at every + ownership boundary. +- The runtime dependency fingerprint is the canonical tuple of producer + identity, artifact kind, complete schema identity, media type, and content + digest. It must be added to every extract, merge, normalize, and validator + checkpoint dependency that receives the generated slot. -Protect deterministic shared input rendering and confirm the existing prompt -assets prepare successfully before changing prompt composition. +## Stage 1: Configuration, Contracts, And Resolution -### Implementation +### Objective -1. Retain one offline prompt preparation test in each extractor package. Each - test should prove registration succeeds, the prompt selects its package-owned - response schema, required dynamic inputs render, and no provider call or - credentials are required. Do not assert total message count, shared-prefix - length, or the complete rendered prompt. -2. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one - table-driven identity test covering: - - identical source material and references produce deeply equal common input - materials; - - reference item insertion order does not change rendered reference bytes; - - `roster` fallback produces the canonical `party` material; - - an explicit non-empty `party` wins over `roster`; and - - missing optional slots render the existing single-space placeholder. -3. Do not add a full HTTP/provider test for cross-module prompt equality. - Scriptorium owns provider request serialization, and shared asset ownership - removes the duplicated content that would otherwise need equality testing. +Accept and fully validate ordered steps and structured artifact reference +sources while preserving legacy single-step configuration. No runner behavior +changes belong in this stage. -### Verification +### Changes -Run: - -```sh -go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns -``` - -### Completion Criteria - -- Shared input tests protect deterministic serialization and canonical slot - handling. -- Each extractor's existing behavior-level asset test prepares successfully - without pinning message count or prefix boundaries. -- No production behavior changes in this stage. - -## Stage 2: Make Prompt Asset Manifests Exact - -### Goal - -Use one explicit asset manifest for prompt mounting and prompt fingerprinting, -and stop fingerprinting shared assets a prompt does not render. - -### Implementation - -1. Replace the broad `sharedPromptFiles`, `CommonHashParts`, and - `ReferenceHashParts` grouping in `internal/modules/dnd/shared/assets.go` - with an explicit manifest abstraction: - - ```go - type PromptAssetManifest struct { - ModuleDir string - ModuleFiles []promptfs.ModulePromptFile - SharedFiles []string - } - ``` - - Exact formatting may follow `gofmt`, but retain these fields and meanings. -2. Give the manifest two operations: - - `PromptFS(moduleFS fs.FS) (fs.FS, error)`, which resolves only the named - shared files and delegates composition to `promptfs.ModulePromptFS`; and - - `Hash(moduleFS fs.FS) (string, error)`, which hashes the same module and - shared files in manifest order through `llm.HashAssets`. -3. Keep the shared-name-to-embedded-path mapping private to the shared package. - Reject unknown, duplicate, empty, or path-containing shared names. Return - fresh slices so callers cannot mutate package state. -4. In each `scriptorium_assets.go`, declare one package-local manifest that - includes its YAML definition and every Markdown file referenced by that - definition. Keep ordering stable within the module and shared lists. Use - that manifest for both `RegisterPromptAssets` and - `scriptoriumPromptMetadata`. -5. Migrate the scene chunker to the same API because it uses the shared prompt - helper. Its behavior and prompt ordering remain unchanged in this stage. -6. Mount only shared assets actually referenced by each prompt. Before Stage 3: - - NPC and scene prompts must not include or hash `common-dnd-npcs.md`; - - spell and combat prompts must include and hash it; and - - all four prompts must include and hash the shared system, transcript, and - campaign-reference files they render. -7. Retain the response-schema fingerprint as its existing independent - fingerprint. Do not include response schema bytes in the prompt manifest. +1. Add an ADR under `docs/adr/` following the existing numbering and template. + Record the bounded ordered-step extension, the single pipeline-wide + input/chunk/output boundary, explicit generated bindings, barriers between + steps, and the rejection of a general DAG. +2. In `internal/core/config/file_config.go` and its validation/application + helpers: + - add file forms for ordered steps and discriminated reference sources; + - use strict YAML decoding for the mapping form and reject unknown fields, + empty selectors, ambiguous scalar-plus-artifact values, and non-string + scalar paths; + - reject pipelines containing both `artifacts` and `steps`, empty explicit + steps, duplicate trimmed step IDs, and duplicate lane IDs across steps; + - preserve list order for steps and validator chains and deterministic key + ordering for lane maps; and + - update cloning, defaults application, effective-config rendering, and + redaction so selectors remain visible but external paths follow the current + path redaction policy. +3. In `internal/framework/contracts/contracts.go`, add generated artifact-kind + compatibility to `ReferenceSlot` and update `CloneReferenceSlots` to deep + copy it. Keep existing external-slot behavior unchanged. +4. In `internal/framework/pipeline/profile.go` and reference-resolution helpers: + - introduce the canonical step and reference-source types described above; + - normalize legacy `artifacts` to resolved step `default`; + - resolve all step lanes and module/validator contracts before resolving + generated bindings, so producer codec metadata and consumer slots are + available for compatibility checks; + - expand a step-scoped binding to every selected target in that step that + declares the slot, for both external and generated sources; do not treat + targets without that slot as errors; + - retain current pipeline, target-local, and CLI precedence for external + bindings, inserting step-local external bindings between pipeline defaults + and target-local bindings; + - reject every generated/external collision on an effective target slot and + every step-scoped/target-local generated collision; + - reject missing producers, same-step or forward producers, undeclared + target-local slots, non-normalized producer lanes, incompatible kinds, + schemas or media types, and selectors made ambiguous by duplicate IDs; and + - include step IDs/order, lane membership, producer selectors, expanded + consumers, and module/validator policy in canonical cloning and the + pipeline digest. +5. Migrate catalog, config validation, `--only` lane selection, checkpoint + identity construction, reference provenance discovery, and debug/effective + configuration code to iterate the resolved step model. Reject `--only` when + the source profile has explicit steps; retain its current behavior for the + implicit `default` step. ### Tests -- Replace broad shared asset tests with table-driven manifest tests for valid - composition, exact mounted files, unknown names, duplicate names, invalid - names, missing module files, missing shared files, and defensive copying. -- Independently construct the expected `llm.AssetHashPart` list in manifest - tests and assert that `Hash` equals `llm.HashAssets` over exactly that list. - This proves unused shared assets are excluded and listed assets participate - without adding mutation hooks for the embedded filesystem. -- Keep one package-level registration test per prompt; remove redundant - per-file mounting assertions when the shared manifest tests already own that - behavior. +- Extend `internal/core/config/*_test.go` with table-driven contracts for legacy + shorthand, explicit step order, strict source-form parsing, mutual exclusion, + identity collisions, cloning, effective output, and redaction. +- Extend `internal/framework/pipeline/profile_test.go`, + `typed_resolution_test.go`, and reference tests for deterministic expansion, + precedence/conflicts, backward/forward rules, global lane uniqueness, + generated-kind/media compatibility, canonical cloning, and digest changes. +- Add a contract test showing a legacy profile resolves to `default` with an + otherwise equivalent lane lifecycle and ordering. +- Prefer one representative case per validation owner; do not duplicate every + parser failure at the resolver layer. -### Verification +### Completion Gate -Run: +Run focused config, contracts, and pipeline resolution tests, then `go test +./internal/core/config ./internal/framework/contracts ./internal/framework/pipeline`. +The runner may still reject or lack execution support for more than one +resolved step, but all configurations and topology must resolve without a flat +parallel lane model. -```sh -go test ./internal/framework/promptfs ./internal/modules/dnd/shared ./internal/modules/dnd/chunk/scenes ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns -``` +## Stage 2: Step-Aware Preparation And Execution -### Completion Criteria +### Objective -- Mounting and hashing are driven by the same ordered manifest. -- No prompt fingerprints an unused shared asset or omits a rendered asset. -- Prepared prompt messages are unchanged from Stage 1. +Execute independent ordered steps with hard barriers, even when no generated +references are configured. Preserve the existing typed lane lifecycle and one +bounded run-wide concurrency budget. -## Stage 3: Factor And Reorder Shared Prompt Messages +### Changes -### Goal - -Implement the target prompt layout defined above using canonical shared assets -and maximize the reusable all-lane and spell/combat content. - -### Implementation - -1. Add these embedded assets under - `internal/modules/dnd/shared/assets/prompts`: - - `common-dnd-extraction-evidence.md`; - - `common-dnd-identity.md`; and - - `common-dnd-immediate-resolution.md`. -2. Write their content according to the fixed semantic boundaries in - `Target Prompt Layout`. Use no template inputs in the identity or immediate - assets. The evidence asset also needs no new input; it refers generically to - the transcript and references already presented later. -3. Update the three extractor YAML files to use the exact order, roles, and - cache-control values in the target table. Do not change prompt IDs, versions, - default profiles, input declarations, output schema paths, or repair counts. -4. Update each package's prompt manifest from Stage 2 so it lists exactly the - new shared dependencies in rendered order. -5. Delete duplicated policy prose from local `task.md` and `instructions.md` - files while retaining every module-specific rule. Preserve one and only one - `Return exactly one JSON object` rule through the common evidence message. -6. Keep the spell catalog input and NPC registry input byte generation - unchanged. Keep the NPC extractor free of an `npcs` input. -7. Update package asset tests only for behavioral wiring: prompt preparation, - response schema selection, and required dynamic input rendering. Do not add - assertions for total message count, exact shared-prefix length, or the end - index of a shared section. The shared manifest tests own canonical asset - selection. -8. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same - change to describe the implemented common-prefix composition, exact prompt - manifest fingerprinting, and cache-control placement. Describe current - behavior, not this staged plan. - -### Verification - -Run: - -```sh -go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns ./internal/modules/dnd/register -go test ./internal/framework/llm -``` - -### Completion Criteria - -- Shared rules exist in one embedded file and have no local paraphrased copy. -- Every applicable prompt references those shared assets in the documented - order and with the documented cache-control policy. -- Prompt fingerprints change for the intentional prompt contract change and - include every newly rendered asset. -- No artifact schema, durable representation, module selection, or reference - contract changes. - -## Stage 4: Centralize Chunk Prompt Material - -### Goal - -Remove the three identical `chunkSourceInput` implementations and make common -transcript input preparation a single D&D-owned behavior. - -### Implementation - -1. Add `internal/modules/dnd/shared/extraction_inputs.go` with: - - ```go - func ChunkPromptMaterial(req contracts.TypedExtractionRequest) (contracts.LLMInputMaterial, error) - ``` - -2. Preserve the existing behavior exactly: clone `req.SourceInput`; fall back - to chunk content and media type when content is empty; require content bytes - to equal `req.Chunk.Content`; default the material name to `source`; default - media type from the chunk; and populate `SizeBytes` when zero. -3. The helper may assume the caller has already checked `req.Chunk != nil`. - Return a D&D-shared error without an extractor name. Each extractor wraps it - with its existing `extractorErrorf`, retaining module context. -4. Replace all three local helpers and remove now-unused `bytes` imports. -5. Do not centralize the remaining request checks. Their typed result handling - and module-specific errors make the small duplication clearer than a - callback- or generic-heavy abstraction. +1. Refactor `internal/framework/pipeline/prepare.go` so `Prepare` walks resolved + steps in order and constructs every lane module and validator before it + returns. A failure in any later step must occur before source parsing. + Materialize only external reference sources during this phase; generated + selectors carry no bytes yet. +2. Refactor `runner.go` and `runner_concurrent.go` into a small ordered-step + coordinator plus the existing bounded lane engine: + - parse input and compute/reuse the chunk plan once; + - invoke the lane engine once per step, passing only that step's prepared + lanes and immutable request-reference view; + - wait for every lane in the step to become terminal before advancing; + - reuse the same worker/provider limits without permitting tasks from + adjacent steps to overlap; and + - encode output once after all steps succeed. +3. Accumulate accepted, rejected, debug, and checkpoint outcomes across steps. + Sort public results by step index, lane ID, source ID, and chunk index/ref as + applicable. Add step identity to internal errors and debug events where lane + identity alone no longer explains execution context. +4. Preserve current fail-fast cancellation and bounded drain behavior. A + framework error in a step cancels its started work, prevents all later steps, + and prevents output encoding while retaining completed upstream outcomes. +5. Until Stage 3 lands, reject execution of any pipeline with a generated + binding before source parsing. This temporary guard prevents an accepted + configuration from running a consumer without its declared dependency. ### Tests -- Add one table-driven shared helper test covering fallback, clone isolation, - mismatch, default fields, and preservation of explicit metadata. -- Remove duplicate extractor tests only when the shared test fully owns the - behavior. Retain one extractor-level test per lane proving helper errors are - wrapped with that module's context. -- Keep existing tests proving all three extractors pass equal common prompt - inputs to the LLM contract. +- Add runner tests with deterministic fake modules proving strict barriers, + concurrency within one step, reuse of the same global budget, stable output + and failure order despite inverted completion timing, and no output encoding + after failure. +- Add a preparation test proving every module and validator in every step is + constructed before the input adapter is invoked. +- Retain existing single-step concurrency and cancellation tests unchanged + where possible; migrate fixtures to the implicit `default` resolved step. + +### Completion Gate + +Run `go test ./internal/framework/pipeline ./internal/cli` plus the repository's +race-test target for the framework if one is defined in `docs/development.md`. +Both legacy pipelines and explicit pipelines without generated references must +run successfully. + +## Stage 3: Canonical Generated-Artifact Handoff And Provenance + +### Objective + +Make accepted normalized output from an earlier lane available as an immutable +operation-time reference to later consumers, with safe checkpoint dependencies +and bounded provenance. + +### Changes + +1. Add a domain-neutral handoff component in `internal/framework/pipeline` that: + - locates the producer's accepted normalized output after its step barrier; + - requires exactly one output and rejects missing, rejected-only, or multiple + outputs deterministically; + - canonicalizes it through `ArtifactCodecRegistry.Encode` (decoding a reused + serialized checkpoint through the registered codec first when necessary); + - verifies kind, exact codec schema identity, accepted media type, maximum + size, and one-item cardinality for every expanded consumer slot; and + - creates independently cloned `ReferenceItem` values for fan-out targets. +2. Build all generated bindings required by a consumer step before starting any + lane in that step. If any handoff fails, return one contextual dependency + error and start none of the step's consumers. An accepted typed empty list is + valid; absence of an accepted artifact is not. +3. Merge generated items with each target's already-materialized external + `ReferenceSet` only after resolution has proven there is no collision. Pass + the resulting cloned set through existing extraction, merge, normalize, and + validation request structs. Never mutate prepared static reference sets. +4. Add the canonical generated dependency fingerprint to checkpoint loader and + recorder inputs for every receiving operation immediately in this stage. + This is required before generated pipelines can safely use resume; do not + defer it to selective recomputation work. +5. Extend artifacts and runtime provenance: + - add step ID to artifact-lane, normalized-output, rejected-output, and + reference provenance records where needed for unambiguous context; + - record external and generated origins distinctly; + - for generated references record producer identities, codec schema + identity, media type, digest, and size, but never content or a fabricated + filesystem URI; and + - update generic JSON output and debug summaries to serialize these bounded + fields deterministically. +6. Ensure checkpoint, debug, and output clones own their byte slices and maps. + A consumer or test mutation must not affect another fan-out consumer or the + producer artifact. + +### Tests + +- Add handoff tests for one producer fan-out, empty typed collections, missing + output, rejected-only output, multiple outputs, type/schema/media/size + mismatch, target immutability, and failure before consumer start. +- Add a resume-oriented runner test proving changed canonical producer content + changes the consumer dependency fingerprint and prevents stale reuse. +- Extend artifacts/JSON/debug contract tests with field assertions for bounded + generated provenance and explicit assertions that content and local paths are + absent. Avoid whole-document snapshots. + +### Completion Gate + +Run focused pipeline, artifacts, JSON output, and debug tests, followed by `go +test ./internal/framework/... ./internal/core/artifacts +./internal/modules/generic/output/json`. A generic fake-codec pipeline must +complete a two-step handoff both from fresh execution and a compatible reused +producer checkpoint. + +## Stage 4: Dependency-Aware Checkpoints And Selective Recompute + +### Objective + +Complete safe reuse, transitive invalidation, observable decisions, and the +`--recompute-step` operator control. + +### Changes + +1. Update checkpoint identity and manifests in + `internal/framework/checkpoint`: + - introduce workspace schema `notarius.workspace.v3` and retain explicit + recognition of v1/v2 as incompatible cold misses; + - include step ID in lane-stage manifests and lookup/recording context; + - include ordered topology in the persistent pipeline identity; and + - preserve the Stage 3 generated fingerprint tuple exactly, without content + or secrets in decision reasons. +2. When loading a normalized producer checkpoint, validate its manifest, + deserialize through the registered codec, re-encode canonically, and compare + schema/media/content fingerprints before exposing it to handoff. Corrupt, + missing, rejected, or incompatible state is a cold miss for ordinary resume + and must never reach a consumer. +3. Build a lane-level dependency index from resolved generated bindings. Use it + to invalidate only transitive consumer lanes when producer identity or bytes + change; unrelated lanes, including unrelated lanes in a later step, remain + eligible for reuse. The step barrier still applies when reused and executed + lanes coexist. +4. Add a single-value `--recompute-step ` flag in + `internal/cli/run.go`. Reject repeated occurrences, unknown steps, use + without checkpoint recording, use without `--resume`, and combination with + `--only`. Accept the stable implicit step ID `default`; selecting it forces + every lane in that single step. One selected step is sufficient for this + scope; do not add multi-selection semantics. +5. Convert the selected step to a force-execution set containing every lane in + that step plus the lane-level transitive dependency closure. Loader policy, + not persistent checkpoint identity, applies the force set. Required + predecessors and unrelated lanes remain reusable. If a required predecessor + was not selected and has no reusable accepted artifact, fail before starting + a dependent lane rather than implicitly recomputing it. +6. Replace the boolean-only checkpoint reporting model as needed with a bounded + decision category: `executed`, `reused`, `forced_recompute`, or + `dependency_invalidated`, plus a stable reason code and optional safe detail. + Propagate it to checkpoint events, manifests, debug summaries, and CLI + diagnostics. Do not make exact prose part of a test contract. + +### Tests + +- In checkpoint loader/recorder tests, cover v1/v2 cold misses, v3 step + identity, compatible producer decode, corruption, exact dependency match, + content/schema changes, and bounded reason fields. +- In pipeline tests, cover transitive invalidation and reuse of unrelated work. +- Add one CLI contract table for valid recomputation and every invalid flag + combination, plus one execution test proving the selected closure is forced + while a predecessor and unrelated lane are reused. +- Do not assert internal loader call counts when observable decision records + and outputs establish the behavior. + +### Completion Gate + +Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline +./internal/cli`, then checkpoint/debug state tests under the race detector as +directed by `docs/development.md`. Manually inspect one test fixture's decision +records to confirm they contain no reference bytes, secret values, or local +paths. + +## Stage 5: D&D NPC-First Production Adoption + +### Objective + +Adopt the platform in the production D&D composition: NPC normalization runs +first, and its canonical artifact grounds spell extraction, combat-turn +extraction, and combat-turn normalization at operation time. + +### Changes + +1. Declare the normalized NPC-list artifact kind on every `npcs` consumer slot: + - `internal/modules/dnd/extract/spells`; + - `internal/modules/dnd/extract/combatturns`; and + - `internal/modules/dnd/normalize/combatturns`. + Keep the slot optional for external standalone use; a configured generated + binding becomes required through the framework dependency. +2. Refactor `internal/modules/dnd/npcs/registry` to provide a package-owned, + concurrency-safe operation-time resolver/cache: + - validate and seed construction-time external references so bad static + configuration still fails before source parsing; + - resolve the effective `npcs` item from each operation request; + - reuse the seeded immutable registry when its digest matches and cache a + generated registry by canonical digest so concurrent chunk operations do + not repeatedly decode it; and + - return cloned or immutable views and never retain caller-owned content. +3. Change the spell extractor, combat-turn extractor, and combat-turn + normalizer to obtain the NPC registry from `req.References` for each + operation through that resolver. Do not reconstruct modules at a step + boundary. Retain current behavior when the request has no NPC item. +4. Keep static prompt/schema metadata and construction-time external reference + fingerprints intact. Record generated NPC identity through framework + reference provenance and dependency fingerprints; do not place + operation-varying generated digests into singleton module metadata. +5. Preserve prompt ordering and evidence policy. The generated NPC listing is + stable contextual input before the variable transcript, but it cannot prove + that a spell or combat event occurred. Source-unit citations remain the only + event evidence. +6. Add a maintained explicit two-step D&D configuration under `examples/` and a + matching production-catalog fixture under + `internal/modules/integration/testdata/`. Replace or supplement the two + manual NPC-to-spell and NPC-to-combat configurations with this single + NPC-first workflow; retain standalone examples that demonstrate external + NPC references where they remain useful. +7. Add one offline integration test using scripted/fake LLM responses. Assert + NPC normalization completes first, the exact canonical NPC digest reaches + all three consumers, spell and combat lanes may both succeed, combat + normalization uses the registry, and generated NPC context is not accepted + as source evidence by itself. + +### Tests + +- Add focused registry and module tests for static fallback, generated override, + concurrent cache safety, malformed runtime content, and ownership isolation. +- Extend prompt-input tests only for semantic placement and content. Do not add + exact prompt-length or message-count change detectors. +- Run the single end-to-end offline D&D handoff test rather than duplicating it + separately for every consumer. + +### Completion Gate + +Run all D&D package tests and integration tests, including the race detector for +the registry cache, then parse and resolve every maintained example through the +production catalog. No test may require credentials or a live LLM provider. + +## Stage 6: Current Documentation And Release Verification + +### Objective + +Document the now-implemented behavior in its canonical current owners, retire +obsolete manual workflow guidance, and perform repository-wide verification. + +### Changes + +1. Update `docs/config.md` with version-3 `steps`, legacy `artifacts` + compatibility, reference source forms, precedence/conflict rules, global + identity rules, and the exact D&D two-step example. +2. Update `docs/cli.md` with `--recompute-step`, prerequisites, closure + semantics, invalid combinations, and checkpoint decision categories. +3. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and + `docs/internal/overview.md` with the fixed ordered-step model, preparation + timing, handoff boundary, codec use, provenance, and checkpoint dependency + behavior. Keep the architecture description linear and explicitly state that + this is not a general DAG. +4. Update `docs/operations.md` with resume/recompute procedures and safe failure + diagnosis. Update the D&D integration documents for generated NPC context, + evidence limitations, and the maintained workflow. Update + `docs/policy/testing.md` only if the implementation exposes a genuinely new + durable testing policy; do not restate feature-specific tests there. +5. Update maintained example checks and any README/index links that point to + replaced sequential examples. Keep the feature roadmap as historical target + state until the project applies its normal roadmap-completion process; do not + turn it into a duplicate current-behavior manual. +6. Review changed exported identifiers and package comments, run formatting and + static analysis, and remove obsolete flat-lane compatibility helpers, + temporary test adapters, dead sequential-workflow code, and TODOs introduced + during earlier stages. ### Verification -Run: +Run the repository-prescribed commands from `docs/development.md`, including: -```sh -go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns -``` +1. all unit and integration tests; +2. race tests for concurrency-sensitive framework, checkpoint, and D&D registry + packages; +3. `go vet` and the normal build; +4. maintained-example/config validation; and +5. documentation checks. -### Completion Criteria +Perform one final acceptance review against every bullet in +[Ordered Pipeline Steps](ordered-pipeline-steps.md). Inspect a successful fresh +run, a resumed run, a forced-recompute run, and a missing-producer failure for +deterministic ordering, correct dependency decisions, bounded provenance, and +absence of generated content or secrets in manifests/debug state. -- Only one production implementation prepares chunk prompt material. -- Extractor behavior and public errors retain useful module context. -- Prompt preparation and shared input behavior tests still pass. +### Completion Gate -## Stage 5: Share Cited Source Traversal - -### Goal - -Give all relatedness validators one deterministic implementation for resolving, -ordering, and deduplicating cited source units while keeping matching semantics -artifact-specific. - -### Implementation - -1. Add `internal/modules/dnd/shared/citations.go` with: - - ```go - func CitedText(doc *source.SourceDocument, refs []source.SourceRef) (string, error) - ``` - -2. Validate every range with `source.ValidateRef`. Resolve ranges against - document order, include each covered source unit once even when ranges - overlap, and join included unit text with a single newline. Return an error - for a nil document or any invalid range. Return `"", nil` for an empty ref - slice with a non-nil document. Do not mutate the document or refs. -3. Add table-driven tests for nil documents, empty refs, invalid source IDs, - unknown/reversed unit IDs, disjoint ranges supplied out of order, adjacent - ranges, and overlapping/duplicate ranges. Output must always follow document - order. -4. Replace spell, NPC, and combat relatedness validators' local cited-text - traversal with `shared.CitedText`. -5. When shape is invalid or `CitedText` returns an error, relatedness validators - approve without relatedness warnings so shape/source-reference validators - remain the sole owners of those defects. -6. Keep matching local: - - spells compare the canonical spell name case-insensitively against combined - cited text; - - NPCs use `identity.ComparisonKey` for names and aliases; and - - combat uses its existing comparison-key actor logic and declaration token - heuristic. -7. Strengthen matching tests with Unicode/apostrophe variants, multiword names, - overlapping ranges, and a short-name substring false-positive case. For - names and actors, require token/word-boundary-aware matching rather than raw - substring matching. For multiword values, match the consecutive normalized - token sequence. Keep the combat declaration rule of at least one normalized - token of four or more runes. -8. Put shared normalized tokenization and consecutive-token matching in - `internal/modules/dnd/shared` only if at least two validators use it after - the change. Otherwise leave the matching helper local; do not create a - single configurable matching engine. - -### Verification - -Run: - -```sh -go test ./internal/modules/dnd/shared ./internal/modules/dnd/validate/spells/source_relatedness ./internal/modules/dnd/validate/npcs/source_relatedness ./internal/modules/dnd/validate/combatturns/source_relatedness -``` - -### Completion Criteria - -- Cited range validation, ordering, overlap handling, and text assembly have - one production owner. -- Relatedness remains warning-only and ignores invalid prerequisite data. -- Artifact-specific matching policies remain understandable in their validator - packages. - -## Stage 6: Align Validator Policy And Diagnostics - -### Goal - -Make validator checkpoint identity, prerequisite handling, and diagnostic -bounding consistent without changing validator-chain order or durable reason -codes. - -### Implementation - -1. Add these local policy constants and `CheckpointFingerprintProvider` - implementations: - - spell shape: `dnd.spells.validator.shape.v1`; - - spell source refs: `dnd.spells.validator.source_refs.v1`; and - - spell source relatedness: `dnd.spells.validator.source_relatedness.v1`. -2. Do not add a separate policy fingerprint to the spell catalog validator; its - effective catalog digest remains its existing semantic checkpoint identity. - If its non-catalog validation policy changes during this work, add a second - `policy` fingerprint rather than replacing `effective_catalog`. -3. Change spell source-reference validation to approve when spell shape is - invalid, matching NPC and combat prerequisite behavior. -4. Change spell source-reference validation to collect all reference issues, - truncate individual errors with `shared/diagnostics.Truncate`, and return a - bounded aggregate with `shared/diagnostics.Aggregate`. Preserve - `invalid_source_refs`. -5. Confirm NPC and combat source-reference validators follow the same - prerequisite and bounded-aggregate policy; refactor only enough to share - obvious local structure. Do not introduce a generic typed validator builder. -6. Keep shape validators responsible for malformed artifact fields and source - reference validators responsible for document/range validity. -7. Add or consolidate package-level tests for policy fingerprints, invalid-shape - deferral, aggregation of multiple invalid refs, bounded diagnostics, strict - options, registration, and deterministic execution class. Prefer a small - table of behavioral expectations in each typed package over a reflection- - based cross-package harness. -8. Update `docs/internal/modules.md` to describe the aligned prerequisite and - checkpoint policy after it is implemented. - -### Verification - -Run: - -```sh -go test ./internal/modules/dnd/validate/... -go test ./internal/modules/dnd/register -``` - -### Completion Criteria - -- Every deterministic validator policy affecting checkpoint reuse has an - explicit semantic fingerprint. -- Later validators do not duplicate shape rejection. -- All source-reference diagnostic output is bounded. -- Existing reason codes and validator-chain order are unchanged. - -## Stage 7: Clarify D&D Registration And Naming - -### Goal - -Make production composition easier to audit without introducing heterogeneous -generic descriptors or changing registration behavior. - -### Implementation - -1. Keep package `internal/modules/dnd/register`, but split its current concerns - into focused files: - - `register.go`: public `Register`, registry validation, and ordered execution - of named registration functions; - - `modules.go`: codecs, chunker, extractors, mergers, normalizers, no-op - normalizers, and prompt asset registrations; - - `validators.go`: production and generic test validator registrations; - - `chains.go`: the six default extract/normalize chain mappings; and - - `merge.go`: typed append functions and deep-clone helpers. -2. Use small private functions such as `registerModules`, - `registerValidators`, `registerPromptAssets`, and - `registerDefaultChains`. Keep the existing ordered `registration{name, - register}` error-context pattern within each group. -3. Do not create one slice containing generic lane descriptors; Go cannot retain - the heterogeneous typed codec and module relationships there without - erasure or callbacks that obscure more than they clarify. -4. Keep append and clone behavior in the `register` package for this change. - Moving it would require a new lane-ownership package with no independent - domain responsibility. -5. Normalize import aliases in the registrar to the pattern - `spellextract`, `npcextract`, `combatextract`, `spellnormalize`, - `npcnormalize`, and `combatnormalize`, with corresponding validator aliases. - This is internal naming only. -6. Preserve registration order, error prefixes, default chain contents and - order, reference-slot/spec behavior, and prompt asset collection. -7. Update `register_test.go` only as required by file movement. Tests should - continue asserting observable registry contents and chain policy, not the - new private helper call graph. - -### Verification - -Run: - -```sh -go test ./internal/modules/dnd/register -go test ./internal/modules/dnd/... -``` - -### Completion Criteria - -- Production composition is grouped by responsibility and remains explicit. -- The refactor produces no registry, chain, capability, or error behavior - change. -- Tests do not couple to private registration helpers. - -## Stage 8: Final Integration, Documentation, And Evaluation - -### Goal - -Verify the complete target state, update canonical current-behavior documents, -and gather quality/cache evidence without making live services part of the -default test suite. - -### Implementation And Review - -1. Re-read `docs/roadmap/dnd.md` and verify every completion criterion against - production code and tests. Do not mark an item complete based only on this - implementation plan. -2. Review naming across the three lanes. Harmonize internal aliases and private - policy constant names, but do not rename compatibility-sensitive identifiers - listed in `Fixed Decisions And Constraints`. -3. Review prompt and durable schemas for accidental duplication. Make no schema - refactor unless composition already exists and the change is behavior-free; - the fixed decision for this plan is to leave them package-owned and separate. -4. Consolidate redundant tests created by intermediate stages. Retain: - - one shared manifest test suite; - - behavior-level prompt preparation tests without prefix-length snapshots; - - focused package tests for artifact-specific prompts and validators; and - - existing production registration contract coverage. -5. Update `docs/internal/modules.md` and `docs/internal/llm.md` so they are the - canonical description of the final implemented behavior. Remove superseded - implementation details rather than appending a second description. -6. Update `docs/roadmap/dnd.md` to record implementation status. Remove completed - future-work details that are fully owned by current internal documentation, - leaving only genuinely deferred outcomes. Do not turn the feature roadmap - into a second current-behavior reference. -7. If credentials and the maintained human-reviewed transcript set are - available, run an explicitly opt-in comparison using identical profiles and - inputs before and after the prompt change. Record only aggregate extraction - review results, prompt token counts, cached-token/cache-write metrics, and - non-secret prompt hashes. Never commit transcript content, rendered prompts, - credentials, endpoints, or private reference material. -8. Live evaluation is not a merge gate when credentials or reviewed fixtures - are unavailable. In that case, record the missing external prerequisite in - the remaining roadmap item; do not add a fake cache-hit claim and do not - weaken offline identity tests. - -### Repository Verification - -Run all required checks: - -```sh -go test ./... -go vet ./... -go build ./cmd/notarius -git diff --check -``` - -Also inspect the final diff for: - -- unintended public identifier or durable schema changes; -- prompt rules duplicated between shared and local assets; -- prompt assets rendered but absent from fingerprints, or fingerprinted but - not rendered; -- provider-specific types outside allowed boundaries; -- tests containing real transcript/reference material or credentials; and -- unrelated changes in a pre-existing dirty worktree. - -### Completion Criteria - -- Every completion criterion in the feature roadmap is either implemented and - documented in its canonical current-behavior owner or explicitly retained as - deferred roadmap work. -- All focused and repository-wide checks pass. -- The final test suite protects behavioral contracts without retaining - redundant implementation snapshots. +The feature is complete only when every acceptance criterion in the feature +roadmap is demonstrated by a stable test or maintained example and the full +repository verification suite passes. ## Open Questions -None. The plan fixes all choices required for implementation. Live provider -evaluation may depend on credentials and reviewed fixtures, but that is an -external acceptance prerequisite rather than an unresolved design decision. +None. The feature roadmap and this plan fix all product and architecture choices +required for implementation. If an implementation detail conflicts with a +policy document, the policy document takes precedence; if it would change the +product semantics above, stop and amend the roadmap rather than deciding it in +code. diff --git a/docs/roadmap/ordered-pipeline-steps.md b/docs/roadmap/ordered-pipeline-steps.md new file mode 100644 index 0000000..90f43b4 --- /dev/null +++ b/docs/roadmap/ordered-pipeline-steps.md @@ -0,0 +1,340 @@ +# Proposed Scope: Ordered Pipeline Steps + +## Status + +Proposed as the next implementation scope. This document refines the +near-term priority in [Future Work](future.md) into a bounded feature target. It +defines desired behavior and scope, not a file-by-file implementation plan. + +## Recommendation + +Implement ordered pipeline steps, generated artifact references, and +dependency-aware checkpoint reuse as one coherent platform capability. Prove +the capability with the D&D workflow in which accepted normalized NPC output +is produced first and then supplied to spell extraction, combat-turn +extraction, and combat-turn normalization. + +This should precede item extraction. It establishes the artifact-handoff and +dependency semantics that the planned item lane will also consume, without +combining two substantial features in one implementation effort. + +## Intended Outcome + +A configured pipeline may contain multiple ordered steps while retaining one +pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run +manifest, and failure boundary. Every artifact lane still follows the fixed +extract, validate, merge, validate, normalize, and validate lifecycle. Steps +add explicit barriers between groups of lanes; they do not create arbitrary +stage graphs. + +An accepted normalized artifact from an earlier step may be bound explicitly +to declared reference slots in a later step. The framework remains +domain-neutral, and generated references remain contextual material rather than +source evidence. + +## Fixed Product And Architecture Decisions + +### Pipeline shape + +- Input parsing and chunk planning remain pipeline-wide and execute once. +- A step contains one or more artifact lanes. Step order is configuration order. +- Lanes within a step remain independent and may use the existing bounded + concurrency model. +- A later step cannot begin until every lane in the current step is terminal + and every generated artifact it requires is accepted and available. +- Public artifact and failure ordering is step order followed by deterministic + lane and source-chunk order, never completion order. +- Output encoding occurs once, after every step succeeds. +- This is not an arbitrary DAG, a general workflow language, concurrent + cross-lane reconciliation, or permission for modules to invoke other modules. + +### Configuration model + +Existing single-step pipelines remain valid. A top-level `artifacts` map is +treated as an implicit step with stable ID `default`. A pipeline may configure +either `artifacts` or `steps`, but not both. Explicit steps must be non-empty +and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique +across the entire pipeline so output paths, selectors, manifests, errors, and +checkpoint scopes remain unambiguous. + +The target configuration shape is: + +```yaml +pipelines: + dnd-session: + input: seriatim + chunk: generic + steps: + - id: identify-npcs + artifacts: + npcs: + extract: dnd/npcs + normalize: dnd/npcs + - id: grounded-events + references: + npcs: + artifact: + step: identify-npcs + lane: npcs + artifacts: + spells: + extract: dnd/spells + normalize: dnd/spells + combat: + extract: dnd/combat-turns + normalize: dnd/combat-turns + output: json +``` + +Existing scalar reference values continue to represent external file paths. +The structured `artifact` form identifies accepted normalized output from one +earlier step and lane. Generated artifact bindings are allowed at step scope or +at an individual module target; they are not inferred from module keys, lane +names, slot names, or domain knowledge. + +A step-scoped reference applies automatically to every selected target in that +step that declares the slot. In the example, one `npcs` binding reaches the +spell extractor plus the combat extractor and normalizer. A target-local +binding is used when only one module should consume the artifact. + +Pipeline-level external references remain defaults. Step-local external +references override pipeline-level external defaults, and target-local +external references retain their existing precedence. A generated reference +and an external reference may not resolve to the same effective target slot; +configuration or a runtime override that creates that conflict is invalid. +Likewise, a step-scoped and target-local generated binding cannot both target +the same effective slot. + +Each effective target slot accepts at most one producer. One producer may fan +out to multiple compatible slots in a later step. Aggregating several producer +artifacts into one slot is outside this scope. + +Reference-slot specs gain optional generated-artifact compatibility metadata. +A generated binding is allowed only when the consumer slot declares the +producer's artifact kind; the producer's registered codec supplies the exact +schema identity and media type used for the handoff. The D&D `npcs` consumer +slots declare the normalized NPC-list artifact kind. Existing external-file +slots and bindings retain their current behavior and do not acquire an artifact +kind merely because their bytes happen to decode as one. + +### Resolution and preparation + +Resolution validates the complete ordered structure before source processing. +It must reject duplicate identities, missing producers, same-step or forward +references, undeclared slots, reference conflicts, and incompatible artifact +kind, schema, media type, or cardinality constraints that are statically +discoverable. Size is checked when canonical producer bytes exist at handoff. +Ordered steps make cycles structurally impossible; resolution must not +introduce a general graph scheduler to rediscover their order. + +The resolved pipeline and its digest include step order, step IDs, lane +membership, generated-reference topology, producer identity, consumer targets, +and existing module and validator policy. Cloning, redaction, canonical JSON, +debug summaries, and manifests preserve the same structure without reference +content or secrets. + +All modules and validators are still selected, option-validated, and +constructed before source parsing. Generated content cannot be supplied during +construction because it does not exist yet. The framework therefore augments +the existing operation-request `References` at the step boundary. Consumers +that currently assume an NPC registry is construction-only must accept the +generated registry from their operation request without deferring general +module construction until after upstream work. + +Only validation that inherently depends on generated bytes may occur at the +handoff. A handoff validation failure is a contextual framework error and fails +the run before any consumer in that step begins. + +### Generated artifact handoff + +Only accepted normalized output may cross a step boundary. Raw extraction +responses, rejected artifacts, merge intermediates, and validator diagnostics +cannot be bound as references. + +The framework serializes the producer through its registered canonical artifact +codec and constructs one immutable reference item containing: + +- the declared target slot; +- canonical artifact bytes and media type; +- artifact kind and schema ID, name, version, and schema digest; +- canonical content digest and size; and +- producer pipeline, step, lane, and module provenance. + +A generated binding requires exactly one accepted normalized artifact from its +producer lane. No artifact is a missing dependency, while more than one is a +cardinality error; a typed collection such as an NPC list remains one artifact. +Combining several normalized outputs into one reference is aggregation and is +outside this scope. + +The existing slot contract remains authoritative for accepted media types, +maximum size, and cardinality. Generated content is cloned at ownership +boundaries and never exposed through a filesystem path. Manifests and debug +summaries record identities and bounded provenance, not artifact content. + +Configuring a generated binding makes that dependency required even when the +consumer module declares the underlying slot optional. An accepted artifact +whose domain collection is empty is still a valid artifact and may be handed +off. If the producer has no accepted normalized artifact, the entire run fails +with a deterministic dependency error and no later step begins. + +### Checkpoint reuse and selective recomputation + +Generated references participate in downstream checkpoint dependencies by +artifact kind, complete schema identity, media type, and canonical content +digest. The pipeline digest protects topology; stage dependency fingerprints +protect the exact upstream artifact consumed. The runner must never combine a +new or changed producer with stale dependent output. + +Compatible producer checkpoints may be decoded through the registered codec +and handed to later steps without rerunning the producer. A missing, rejected, +corrupt, incompatible, or changed producer invalidates every transitive +dependent checkpoint. Independent work remains reusable. + +Add one operator control, `--recompute-step `, with these semantics: + +- it requires checkpoint recording and `--resume`; +- the selected step and all transitive dependents execute rather than reuse + their checkpoints; +- valid required predecessors and unrelated work remain reusable; +- the recompute selection affects loader decisions, not the persistent + checkpoint identity of otherwise identical work; and +- the command fails before dependent execution if a required predecessor has + no reusable accepted artifact. + +Existing `--only` behavior remains unchanged for implicit single-step +pipelines. Combining `--only` with explicit multi-step pipelines is outside +this scope and should be rejected with actionable guidance rather than given +implicit dependency-expansion semantics. + +Checkpoint events, manifests, and diagnostics distinguish executed, reused, +forced-recomputed, and dependency-invalidated work. Invalidation reasons are +bounded, deterministic, and free of reference content, local paths, or secrets. +Old checkpoint state need not be migrated; it must produce a safe, explicit +cold miss rather than an error or unsafe reuse. + +### Failure, cancellation, and concurrency + +The existing run-wide worker and provider-call limits apply across every step. +Workers may be reused between steps, but concurrency cannot cross a step +barrier. A framework error cancels started work using the existing bounded +drain behavior and prevents later steps and output encoding. Rejections remain +recorded outcomes, but failure to produce a normalized artifact required by a +generated binding escalates to the run-level dependency error described above. + +The failed manifest retains completed upstream outcomes, step and lane +provenance, rejections, checkpoint events, and the dependency failure without +embedding generated artifact content. + +## D&D Proving Workflow + +The production acceptance workflow has two explicit steps: + +1. `identify-npcs` runs the NPC lane through normalization and its complete + validator policy. +2. `grounded-events` receives the canonical NPC artifact in its step-scoped + `npcs` reference and runs spell and combat-turn lanes. The binding reaches + spell extraction, combat-turn extraction, and combat-turn normalization. + +Spell and combat-turn lanes may execute concurrently after the handoff. NPC +content may ground names and identities but cannot establish a spell cast or +combat event; source units remain the only event evidence. + +The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be +replaced or supplemented by one ordered-pipeline example. Existing module keys, +artifact contracts, reference slot names, prompt IDs, and D&D evidence policy +remain unchanged. + +## Included Work + +- Configuration parsing, validation, cloning, defaults, redaction, and + documentation for explicit steps and structured generated references. +- Domain-neutral resolved step, dependency, producer, and consumer identities. +- Generated-artifact compatibility metadata on reference-slot contracts, + including D&D NPC-list declarations for every `npcs` consumer. +- Step-aware preparation metadata and runner orchestration. +- Canonical codec handoff into existing reference request contracts. +- Required-dependency failure and bounded provenance behavior. +- Dependency-aware checkpoint reuse, invalidation, events, and selective step + recomputation. +- D&D NPC-first production composition for spell and combat-turn consumers. +- Refactoring the affected D&D consumers so generated NPC references are + available at operation time while retaining early static construction. +- Maintained examples and updates to current architecture, configuration, CLI, + operations, internal, integration, and testing documentation when behavior + lands. +- An ADR recording the bounded ordered-step extension to the fixed pipeline + architecture and its explicit rejection of a general DAG. + +## Explicitly Out Of Scope + +- D&D item extraction or any other new artifact lane. +- Cross-artifact NPC ID fields or artifact-schema migration machinery. +- Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or + module-controlled scheduling. +- Multiple source inputs, per-step input adapters, per-step chunk plans, or + per-step output encoders. +- Aggregating multiple generated artifacts into one reference slot. +- Optional or best-effort generated dependencies; a configured dependency is + required in this scope. +- Prior-run or cross-pipeline generated references. +- `--only` dependency closure for explicit multi-step pipelines. +- Cross-lane reconciliation or domain concepts in the generic framework. +- Live-provider tests or model-quality changes to D&D prompts. + +## Acceptance Criteria + +The scope is complete when: + +- all existing single-step configurations retain their current behavior; +- explicit step order and dependency topology resolve deterministically and + affect pipeline identity; +- invalid producer, consumer, conflict, ordering, type, schema, media, and + cardinality configurations fail before source processing when statically + discoverable, while content-size violations fail at handoff; +- no consumer step begins before all required generated artifacts are accepted, + canonicalized, and validated for its target slots; +- one producer artifact fans out safely to every compatible target selected by + a step-scoped binding; +- missing required producer output fails the complete run before dependent work; +- changing NPC output invalidates spell and combat-turn checkpoints while + leaving compatible independent work reusable; +- selective step recomputation executes exactly the selected dependency closure + and reports why work was executed, reused, or invalidated; +- the D&D ordered workflow supplies NPC content to spell extraction, combat-turn + extraction, and combat-turn normalization without treating it as evidence; +- completion timing cannot change public ordering, failure selection, or + dependency behavior; +- output, manifests, checkpoints, and debug artifacts contain the required + identities and provenance without leaking generated reference content; and +- repository-wide tests, vet, build, maintained-example checks, and + documentation validation pass. + +## Testing Strategy + +Tests should protect behavior and invariants rather than the implementation's +internal scheduler shape. + +- Configuration contract tests own legacy shorthand, explicit step parsing, + source-form discrimination, conflicts, and redaction. +- Resolution tests own ordering, global lane uniqueness, dependency validation, + slot compatibility, fan-out, cloning, canonical JSON, and digest changes. +- Runner tests own step barriers, within-step bounded concurrency, stable + ordering, cancellation, required-producer failure, and immutable handoff. +- Checkpoint tests own producer decoding, exact dependency matching, transitive + invalidation, forced recomputation, cold misses, and bounded decisions. +- One CLI contract test should cover the recompute control and its invalid + combinations. +- One D&D integration test with offline fake LLM responses should prove the + complete NPC-to-spell-and-combat handoff, including combat normalization. +- Maintained configuration examples should be parsed and resolved through the + production catalog. + +Do not add scheduler choreography tests, exact goroutine-count assertions, +complete manifest snapshots, exact diagnostic strings, or duplicated tests for +every invalid configuration at every layer. No test may require credentials or +a live model provider. + +## Open Questions + +None required to define this scope. Exact internal type names and implementation +decomposition are intentionally not feature-policy decisions.