notarius/docs/internal/overview.md

Internal Overview

This document inventories the implemented Notarius components. Normative boundaries and dependency direction belong in Architecture; external behavior belongs in the CLI, Configuration, Operations, and integration contracts.

Execution Path

cmd/notarius delegates to internal/cli, the production composition root. The CLI loads configuration, builds the production catalogs and runtime collaborators, invokes internal/framework/pipeline, and places the logical output files returned by the runner. Cache and debug collaborators are supplied at this boundary.

Resolution produces a fixed ordered workflow of steps and globally unique, sorted artifact lanes. Preparation constructs the complete module and validator set before the runner receives source bytes. Source parsing and chunking are serial. Each step then uses a bounded run-wide extraction pool followed by serial per-lane merge and normalize continuations. A step barrier prevents later consumers from starting until all earlier lanes are terminal and their required normalized artifacts have crossed the typed handoff.

Application Boundary

Package	Implemented responsibility
cmd/notarius	Executable entry point and process exit delegation.
internal/cli	Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results.

Core Packages

Package	Implemented responsibility
internal/core/artifacts	Run-manifest and provenance models.
internal/core/config	Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution.
internal/core/debugbundle	Explicit per-run debug-bundle allocation and redacted summary writing.
internal/core/fileio	Generic confined atomic file and JSON writes with caller-selected permissions.
internal/core/source	Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization.

Framework Packages

Package	Implemented responsibility
internal/framework/contracts	Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types.
internal/framework/pipeline	Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population.
internal/framework/validate	Shared validator decision and cardinality helpers.
internal/framework/llm	Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction.
internal/framework/promptfs	Builds module prompt filesystems from module-owned and caller-provided shared prompt assets.
internal/framework/checkpoint	Root-based checkpoint loading, recording, identity, and payload serialization.
internal/framework/chunkplan	Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication.
internal/framework/chunkmap	Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization.
internal/framework/debug	Root-based framework and LLM debug recording.

Framework contracts provide typed artifact, provenance-wrapper, chunk-validator, serialized-validator, and typed-validator interfaces. The runner owns handoff provenance, validation sequencing, rejection handling, checkpoint and debug boundaries, and final manifest assembly.

Artifact registries support heterogeneous typed extraction entries and kind-specific merger, normalizer, and validator variants. Resolution derives a lane's kind from its extractor, requires the matching codec, verifies exact Go type equality across the lane, and records schema identity in the resolved lane and pipeline digest. Registry entries carry separate option-validation and run-local construction closures. Preparation injects shared dependencies and constructs input, chunk, validators, ordered lanes, and output before source parsing. Production modules use strict construction-time option decoding, and LLM-backed modules retain the injected shared client. The D&D family registers the canonical dnd/spell-list, dnd/npc-list, dnd/combat-turn-list, dnd/item-event-list, dnd/npc-interaction-list, and dnd/scene-description-list codecs, typed spell, NPC, combat, item-event, interaction, and scene-description extractors and normalizers, validators, plus kind-specific generic merge strategies; generic JSON validators use the serialized-validation contract. The runner executes lanes through private exact-type-checked closures, coordinates extract results independently of completion timing, and serializes artifacts only through their codec at checkpoint, debug, and output boundaries.

Production Extensions

The canonical catalogs of user-selectable module and validator keys are in Configuration. The implemented module packages are:

Package	Implemented responsibility
internal/modules/seriatim/input/transcript	Parses the supported Seriatim transcript format into the generic source model.
internal/modules/generic/chunk/units	Splits ordered source units by unit count and overlap.
internal/modules/dnd/chunk/scenes	Produces contiguous D&D scene chunks from structured model output.
internal/modules/dnd	Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, combat-turn-list, combat-turn, item-event-list, item-event, NPC-interaction-list, and scene-description-list artifact types.
internal/modules/dnd/codec/spells	Strictly decodes and stably encodes the durable D&D spell-list representation.
internal/modules/dnd/codec/npcs	Strictly decodes and stably encodes the durable D&D NPC-list representation.
internal/modules/dnd/codec/combatturns	Strictly decodes and stably encodes the durable D&D combat-turn-list representation.
internal/modules/dnd/codec/itemevents	Strictly decodes and stably encodes the durable D&D item-event-list representation.
internal/modules/dnd/codec/npcinteractions	Strictly decodes and stably encodes the durable D&D NPC-interaction-list representation.
internal/modules/dnd/codec/scenedescriptions	Strictly decodes and stably encodes the durable D&D scene-description-list representation.
internal/modules/dnd/extract/spells	Maps private structured model output to canonical source-grounded D&D spell lists.
internal/modules/dnd/extract/npcs	Maps private structured model output to canonical source-grounded D&D NPC lists.
internal/modules/dnd/extract/combatturns	Uses exact scene eligibility to select combat chunks, then maps private structured model output to source-grounded D&D combat-turn candidates.
internal/modules/dnd/extract/itemevents	Maps private structured model output to source-grounded D&D item-event candidates.
internal/modules/dnd/extract/npcinteractions	Maps private structured model output to current-source NPC interaction candidates grounded by a required registry.
internal/modules/dnd/extract/scenedescriptions	Maps one private scene description to the current accepted chunk's ID and exact range.
internal/modules/dnd/npcinteractions	Owns interaction occurrence ordering, valid-evidence checks, and exact interaction identity shared by normalization and invariant validation.
internal/modules/dnd/normalize/combatturns	Canonicalizes and orders merged combat turns, applies exact NPC identity matches, and collapses only exact valid-evidence duplicates.
internal/modules/dnd/normalize/itemevents	Trims, source-orders, and removes only exact valid-evidence item-event duplicates.
internal/modules/dnd/normalize/npcinteractions	Canonicalizes required-registry names, orders interaction occurrences, and collapses only exact valid-evidence duplicates.
internal/modules/dnd/normalize/scenedescriptions	Trims, source-orders, and removes only exactly identical scene descriptions while rejecting ID and range conflicts.
internal/modules/dnd/validate/combatturns	Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for the production combat chains.
internal/modules/dnd/validate/itemevents	Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for item-event chains.
internal/modules/dnd/validate/npcinteractions	Provides deterministic shape, registry, source-reference, source-relatedness, and normalized-invariant validation for interaction chains.
internal/modules/dnd/validate/scenedescriptions	Provides deterministic shape, exact extraction attachment, source-relatedness, and normalized-invariant validation for scene-description chains.
internal/modules/dnd/npcs/registry	Resolves validated normalized NPC references into immutable grounding data and exact identity lookup.
internal/modules/dnd/scenedescriptions/registry	Resolves approved scene descriptions into immutable exact-match combat eligibility data without retaining scene prose.
internal/modules/dnd/npcs/identity	Owns Unicode-aware NPC identity, ID derivation, and registry collision validation.
internal/modules/dnd/spells/catalog	Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup.
internal/modules/generic/merge/appendorder	Combines accepted extraction results in chunk order.
internal/modules/generic/normalize/noop	Preserves accepted merged output.
internal/modules/dnd/normalize/spells	Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity.
internal/modules/dnd/normalize/npcs	Deterministically prepares and safely applies document-level LLM-assisted NPC identity consolidation, preserving canonical evidence, order, and diagnostics.
internal/modules/generic/output/json	Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files.

internal/modules/dnd/shared owns reusable D&D prompt fragments, reference declarations, prompt input assembly, document-aware source-reference ordering and canonicalization, and bounded diagnostics under internal/modules/dnd/shared/diagnostics. The shared NPC grounding fragment is mounted for D&D prompts and is owned by this package. Domain-neutral prompt filesystem composition lives in internal/framework/promptfs.

The dnd/npcs/registry package owns the optional npcs registry boundary. External references are strictly decoded and identity-validated during preparation; generated references are decoded and identity-validated at the ordered step handoff. Both paths retain canonical registry JSON for provenance and emit a names-only projection to operation-time spell, combat, and interaction prompts. Combat and interaction normalization use the canonical registry for exact name lookup. The framework records generated identity and bounded producer provenance, while the raw external reference remains independently tracked by pipeline provenance. An absent registry is represented only by the empty prompt value {"npcs":[]}. Spell and combat consumers use this shared boundary without changing their public module contracts. Interaction consumers require it and retain only current transcript references as durable evidence.

Generic validators under internal/modules/generic/validate provide unconditional test decisions, JSON syntax validation, and JSON Schema validation. D&D spell validators under internal/modules/dnd/validate/spells consume the canonical spell-list type directly to provide shape, effective-catalog, source-reference, and source-relatedness decisions.

Production composition is grouped behind package-family registrars, and every implemented production extension uses its domain-first tree:

Package	Implemented responsibility
internal/modules/generic/register	Registers domain-neutral chunk, merge, normalize, output, and validator implementations.
internal/modules/seriatim/register	Registers the Seriatim input adapter.
internal/modules/dnd/register	Registers D&D modules, validators, default validator policy, and prompt/schema assets.

The CLI allocates the framework registries and asset registry, then invokes these registrars in generic, Seriatim, and D&D order.

Implementation details for all production extensions are in Module Internals.

Run-State Components

Surface	Implemented owners	Internal purpose
Durable output	Output module, pipeline runner, and CLI writer	Return logical consumer files and place them for a run.
Cache checkpoints	internal/framework/checkpoint and internal/cli	Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions.
Chunk-plan cache	internal/framework/chunkplan and internal/cli	Persist and select source-addressed plans before framework materialization.
Debug bundles	internal/core/debugbundle, internal/framework/debug, and pipeline instrumentation	Persist redacted summaries and application-owned traces.

Physical layout, cleanup, recovery, and sensitive-data handling are defined in Operations. Concrete modules receive recorder interfaces and request data, not physical state roots.

Focused Documentation

• Pipeline Internals: resolution, execution, validation, retries, checkpoint/debug hooks, and result assembly.
• Module Internals: production modules, validators, assets, registration, and the contributor recipe for adding an extension.
• LLM Runtime: structured completion contracts, Scriptorium adapter, assets, scheduling, profile recording, and redaction.
• Run State Internals: output, cache, debug collaborator composition, and path safety.