diff --git a/assets/dnd/shared/prompts/common-dnd-transcript-windows.md b/assets/dnd/shared/prompts/common-dnd-transcript-windows.md deleted file mode 100644 index a7edfbe..0000000 --- a/assets/dnd/shared/prompts/common-dnd-transcript-windows.md +++ /dev/null @@ -1,6 +0,0 @@ -Selected Dungeons & Dragons gameplay transcript evidence windows are provided -below. They may be incomplete, non-contiguous, or overlapping. Use them to -evaluate candidate identity, but do not treat absence outside these windows as -evidence. - -{{ input "transcript" }} diff --git a/docs/internal/dnd.md b/docs/internal/dnd.md index 950f164..91e660f 100644 --- a/docs/internal/dnd.md +++ b/docs/internal/dnd.md @@ -84,14 +84,15 @@ replace it with a complete profile of the same ID from the configured PromptKit source. Deployment profile selection is documented in [Configuration](../config.md#promptkit-profiles). -The transcript assets have distinct consumers. Scene chunking consumes the -complete-session `common-dnd-transcript-full.md`; extraction prompts consume -the current-chunk `common-dnd-transcript-chunk.md`; and NPC, location, and item -normalization consume `common-dnd-transcript-windows.md` alongside their -candidate collections. Player, party, glossary, and compatible campaign -references provide disambiguating context only when declared by the active -prompt; they never establish evidence. Reference material is canonically -ordered before rendering so equivalent inputs remain stable. +The D&D transcript assets have distinct consumers. Scene chunking consumes the +complete-session `common-dnd-transcript-full.md`, while extraction prompts +consume the current-chunk `common-dnd-transcript-chunk.md`. NPC, location, and +item normalization instead mount the generic semantic-reconciliation +candidate and transcript-window presentation assets. Player, party, glossary, +and compatible campaign references provide disambiguating context only when +declared by the active prompt; they never establish evidence. Reference +material is canonically ordered before rendering so equivalent inputs remain +stable. Extraction prompts render the common system and identity messages first, then cached campaign references and the cached chunk transcript. Evidence policy and @@ -101,10 +102,11 @@ reusable extraction prefix identical while preserving the lane-specific suffix. Scene chunking intentionally uses a different order: system, cached campaign references, uncached module instructions, then the final ephemeral full -transcript. Entity normalization also has its own order: system, uncached -module instructions, ephemeral reconciliation policy, uncached candidates, and -final ephemeral transcript windows. These orders and cache controls are prompt -behavior; change them only through the owning manifest and prompt declaration. +transcript. Entity normalization also has its own order: D&D system, mandatory +generic protocol, ephemeral domain semantic instructions, generic candidate +presentation, and final ephemeral generic transcript windows. These orders and +cache controls are prompt behavior; change them only through the owning +manifest and prompt declaration. ## Evidence, Candidates, And Normalization @@ -134,12 +136,46 @@ canonicalize display values and evidence, use source-document order for stable output, and issue bounded warnings for changes or collapsed duplicates. NPC, item, and location registry normalizers are intentional exceptions: each first produces a deterministic candidate set, then may use a bounded structured-LLM -proposal to reconcile identity groups. The proposal selects supplied -descriptors—names with their candidate source references—not durable IDs. -Request-local candidate keys may support resolution internally, but are never -included in model input or output. Colliding descriptors are ineligible, and -invalid or unusable proposals retain the deterministic result with retry or -fallback diagnostics; the model does not directly replace durable records. +proposal to reconcile identity groups. + +## Semantic Registry Reconciliation + +The three registry normalizers instantiate the domain-neutral +`internal/framework/semanticreconcile` engine with default bounds. Each +eligible candidate receives a contiguous, one-based `candidate_id` for that +request. The model sees that handle, the candidate label and source-free +evidence ranges, plus bounded transcript windows; it returns only duplicate +groups of supplied handles and one supplied canonical handle per group. It +never returns names, evidence, durable IDs, or replacement records. Identical +labels and evidence remain independently selectable because their handles are +distinct. + +The generic core owns the mandatory handle protocol, candidate and transcript +presentation, the private response schema, source-reference validation, +candidate and combined-material limits, structured completion, proposal +assessment, stable group ordering, and typed plan-application mechanics. The +D&D prompt contributes its system message and registry-specific semantic +instructions. The generic registrar registers the shared prompt and schema; +the D&D registrar registers each consuming prompt and the fallback profile. + +Fewer than two eligible candidates skips the LLM without a semantic warning. +An exceeded bound also skips the call and preserves the deterministic +preprocessed registry, adding the registry's bounded fallback warning. Invalid +structured output or discarded proposal groups use the normalizer's existing +retry contract; retry exhaustion preserves the safe deterministic or +partially applied result and emits its bounded fallback warning. Provider, +transport, cancellation, and context-material failures remain execution +errors. + +Application remains typed and registry-owned. All three policies select the +canonical member's normalized display name, union member evidence in source +order, preserve ungrouped records, and derive durable identity only after +consolidation. NPC IDs derive from the final name. Item IDs also derive from +the final name, and a typed guard prevents currency aliases from crossing +denominations or mixing currency with non-currency records. Location IDs +derive from the final name and final evidence, preserving same-name, +parent/child, and distinct physical-place identities. Registry warning scopes, +reason codes, and postconditions remain outside the generic core. ## Generated References And Grounding diff --git a/docs/internal/llm.md b/docs/internal/llm.md index 1ff72ab..410ac38 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -142,12 +142,28 @@ arrangement and its data-only boundary are defined by [ADR-0011](../adr/0011-centralize-llm-assets.md), rather than by this runtime guide. +The generic registrar is the sole production registration owner for the +semantic-reconciliation default prompt and private response schema. The +domain-neutral reconciliation package also exposes only its mandatory protocol +and candidate/transcript presentation files for domain prompt manifests. D&D +registry normalizers mount those files while retaining ownership and hashing +of their D&D system message, semantic instructions, and complete prompt +declaration. The response schema is therefore registered once even though +several typed normalizers select it. + Mounted prompt assets determine a module's fingerprint. The fingerprint hashes only the module and shared files explicitly selected by its manifest, so an unrelated asset does not invalidate a checkpoint. Schema loaders validate JSON, attach identity and digest metadata, make defensive copies, and expose diagnostics without raw schema bytes. +Semantic-reconciliation normalizers extend this identity with the shared +response-schema digest, framework policy version, and complete limit-policy +digest. Their manifest metadata records the same content-free prompt, schema, +policy, and limit identities together with domain identity and normalization +policies. Request-local handles, source material, proposal content, and raw +asset bytes are not checkpoint metadata. + Private response schemas validate a model transport envelope. They are not the durable artifact schema and should not be documented as an external wire contract. Durable formats and compatibility rules remain in the diff --git a/docs/internal/modules.md b/docs/internal/modules.md index ef27db5..a01b6ed 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -44,12 +44,20 @@ capability during preparation and the output boundary owns publication. See the [Published Evidence Context contract](../integrations/evidence-context.md) for the durable result. +An artifact family is broader than a module: it owns the cohesive domain +feature across its artifact type, codec, stage modules, validators, prompt +policy, schemas, identity helpers, and reference projections. An extractor and +normalizer in one artifact family remain independently registered modules in +their respective pipeline stages. This ownership vocabulary does not create a +new registry or change the fixed pipeline. + ## Production Composition Production composition is intentionally split by family: - The generic registrar provides the unit chunker, generic JSON validators, - and JSON output encoder. + JSON output encoder, and shared semantic-reconciliation prompt and response + schema assets. - The Seriatim registrar provides the transcript input adapter. Its external input behavior is defined by the [Seriatim contract](../integrations/seriatim.md). - The D&D registrar provides its codecs, extractors, mergers, normalizers, @@ -60,6 +68,36 @@ The CLI owns the composition that invokes these registrars. A module package may register its own family but must not assemble the CLI or make framework packages depend on production extensions. +## Semantic Reconciliation + +`internal/framework/semanticreconcile` is a domain-neutral strategy used by a +typed normalize module; it is not itself a selectable stage module. A +source-backed artifact-family normalizer projects its deterministic records +into contextual candidates and owned typed record envelopes, supplies its +chosen prompt identity and resolved LLM profile, and constructs an engine with +explicit limits. The core filters invalid evidence, assigns contiguous +request-local integer handles, renders bounded candidate and transcript +materials, invokes the structured-completion boundary, and assesses the +returned duplicate groups into a stable non-overlapping plan. + +The normalizer then applies that plan through a typed `ApplicationPolicy`. The +core preserves ungrouped records, contribution order, and provenance while the +artifact family owns group guards, field and evidence consolidation, durable +ID derivation, retry and fallback presentation, warnings, and postconditions. +Request-local handles do not enter the typed value or durable artifact. Fewer +than two eligible candidates skips model invocation; exceeding a candidate or +combined-material bound preserves the deterministic result under the family's +fallback policy. Provider, transport, cancellation, and context-construction +failures remain execution errors. + +The core supplies a conservative generic prompt and the single private +response schema. A domain prompt may substitute its semantic instructions but +mounts the core-owned protocol and candidate/transcript presentation assets. +Prompt, schema, policy, and limit identities participate in manifest metadata +and checkpoint fingerprints. The generic registrar owns production +registration of those shared assets; a consuming domain registrar owns only +its domain prompt. + ## Adding Or Changing A Module 1. Choose the pipeline stage and the typed artifact boundary. Put external diff --git a/docs/internal/overview.md b/docs/internal/overview.md index aad3aec..5c4531d 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -29,6 +29,7 @@ physical state roots. | Generic models | **internal/core/source**, **internal/core/artifacts**, **internal/framework/contracts** | Source documents and chunks, manifests and provenance, plus typed artifact, reference, validation, output, and structured-completion contracts. | | Pipeline framework | **internal/framework/pipeline** | Registries, profile and reference resolution, typed preparation, validation, retry coordination, ordered execution, handoff, and result assembly. | | LLM and prompt runtime | **internal/framework/llm**, **internal/framework/promptfs** | Provider-neutral structured completions, scheduling, profile recording, prompt assets, schema registration, and credential-shaped-value redaction. | +| Semantic reconciliation | **internal/framework/semanticreconcile** | Bounded source-backed candidate preparation, request-local handle proposals, deterministic assessment, typed plan application, and reconciliation identity metadata; see [Module Internals](modules.md#semantic-reconciliation) and [D&D Module Internals](dnd.md#semantic-registry-reconciliation). | | Embedded LLM content | **assets** | Read-only centralized LLM-facing content, scoped by its consuming package; see [LLM Runtime](llm.md#prompt-and-schema-assets) and [D&D Module Internals](dnd.md#prompt-construction). | | Runtime state | **internal/core/fileio**, **internal/core/debugbundle**, **internal/framework/checkpoint**, **internal/framework/chunkplan**, **internal/framework/chunkmap**, **internal/framework/debug** | Confined atomic files, debug bundles, checkpoint and chunk-plan state, accepted chunk maps, and pipeline-facing debug recording. | | Production extensions | **internal/modules/generic**, **internal/modules/seriatim**, **internal/modules/dnd** | Domain-neutral extensions, Seriatim input support, and D&D extraction families registered into the production catalog. | @@ -49,8 +50,9 @@ the CLI composition boundary. composition, and path safety. - [LLM Runtime](llm.md): structured completion, scheduling, prompt assets, profiles, and secret handling. -- [Module Internals](modules.md): generic extension registration, module - construction, validation, and reference mechanics. +- [Module Internals](modules.md): generic extension registration, artifact + families, module construction, semantic reconciliation, validation, and + reference mechanics. - [D&D Module Internals](dnd.md): shared D&D extractor conventions, generated reference projections, and lane-specific exceptions. Durable D&D and Seriatim data shapes remain in the [integration contracts](../integrations/). diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index b7bd5e1..b515fff 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -24,6 +24,12 @@ DAGs or a general workflow language. Every stage remains explicit; general chunking, merging, or normalization behavior must not be hidden inside an extractor. +A stage module is one configured implementation of one pipeline stage. An +artifact family is the cohesive domain feature that owns an artifact across +the explicit stages and supporting codecs, validators, prompts, identity +rules, and reference projections. Artifact-family ownership does not combine +stages or alter the fixed pipeline. + Input and chunking are pipeline-wide. Each selected artifact lane owns its extract, merge, and normalize stages, and the output stage aggregates the run's lane outcomes. @@ -39,6 +45,12 @@ implementations. Domain-neutral model and framework layers provide reusable policy, contracts, and orchestration. Concrete input, pipeline, output, and validation extensions depend inward on those generic layers. +Semantic reconciliation is one such domain-neutral framework mechanism. It +prepares bounded source context, invokes a shared model-judgment protocol, +validates proposals, and applies safe plans through typed policies supplied by +the consuming artifact family. It does not own domain identity, durable IDs, +warning semantics, or artifact construction rules. + Generic layers must not depend on production extensions. Concrete extensions must not compose the application or take ownership of process behavior. The current packages implementing these layers are inventoried in @@ -186,8 +198,13 @@ or domain-specific prompt logic. When a model selects an application entity, callers must supply a contextual selection and deterministically attach the opaque application identity whenever the selection resolves exactly. Models do not receive or reproduce opaque -application identifiers; [ADR-0012](../adr/0012-resolve-opaque-entity-identifiers-deterministically.md) -records the rationale and limited request-local-label exception. +application identifiers. Semantic reconciliation may instead expose +contiguous, one-based candidate handles that exist only for one request; +deterministic code resolves them before typed application, and they never +become durable identity. This is the approved request-local-label application +of [ADR-0012](../adr/0012-resolve-opaque-entity-identifiers-deterministically.md) +recorded by +[ADR-0013](../adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md). LLM calls and other external operations accept cancellation and respect timeouts. Concurrency control belongs in shared runtime plumbing rather than in diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 5b23a66..3969960 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -24,10 +24,12 @@ not as committed release dates. ## Shared Normalization And Quality Work -The active [Semantic Reconciliation Roadmap](semantic-reconciliation.md) owns -the shared source-backed reconciliation core, request-local candidate protocol, -generic prompt policy, typed adapter boundary, and initial D&D registry -migrations. +The implemented source-backed core and initial D&D registry adoption are +described by [Module Internals](../internal/modules.md#semantic-reconciliation) +and +[D&D Module Internals](../internal/dnd.md#semantic-registry-reconciliation). +The [Semantic Reconciliation Roadmap](semantic-reconciliation.md) retains the +original feature scope; the sections below keep broader extensions deferred. ### Large-Collection Semantic Reconciliation diff --git a/internal/modules/dnd/shared/assets.go b/internal/modules/dnd/shared/assets.go index a7e22f8..47910e0 100644 --- a/internal/modules/dnd/shared/assets.go +++ b/internal/modules/dnd/shared/assets.go @@ -27,7 +27,6 @@ var sharedPromptPaths = map[string]string{ "common-dnd-identity.md": "prompts/common-dnd-identity.md", "common-dnd-transcript-full.md": "prompts/common-dnd-transcript-full.md", "common-dnd-transcript-chunk.md": "prompts/common-dnd-transcript-chunk.md", - "common-dnd-transcript-windows.md": "prompts/common-dnd-transcript-windows.md", "common-dnd-references.md": "prompts/common-dnd-references.md", "common-dnd-npc-registry.md": "prompts/common-dnd-npc-registry.md", } diff --git a/internal/modules/dnd/shared/assets_test.go b/internal/modules/dnd/shared/assets_test.go index 9ade648..76e3f2e 100644 --- a/internal/modules/dnd/shared/assets_test.go +++ b/internal/modules/dnd/shared/assets_test.go @@ -22,7 +22,6 @@ func TestPromptAssetManifestPromptFS(t *testing.T) { "common-dnd-system.md", "common-dnd-transcript-full.md", "common-dnd-transcript-chunk.md", - "common-dnd-transcript-windows.md", }, } @@ -51,7 +50,6 @@ func TestPromptAssetManifestPromptFS(t *testing.T) { "assets/prompts/dnd.test/sharedassets/common-dnd-system.md", "assets/prompts/dnd.test/sharedassets/common-dnd-transcript-full.md", "assets/prompts/dnd.test/sharedassets/common-dnd-transcript-chunk.md", - "assets/prompts/dnd.test/sharedassets/common-dnd-transcript-windows.md", } { content, err := fs.ReadFile(fsys, path) if err != nil {