notarius/docs/internal/overview.md

# Internal Overview

This document inventories the implemented Notarius components. Normative
boundaries and dependency direction belong in
[Architecture](../policy/architecture.md); external behavior belongs in the
[CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [integration contracts](../integrations/).

## Execution Path

`cmd/notarius` delegates to `internal/cli`, the production composition root.
The CLI loads configuration, builds the production catalogs and runtime
collaborators, invokes `internal/framework/pipeline`, and places the logical
output files returned by the runner. Cache and debug collaborators are supplied
at this boundary.

Resolution produces a fixed ordered workflow of steps and globally unique,
sorted artifact lanes. Preparation constructs the complete module and validator
set before the runner receives source bytes. Source parsing and chunking are
serial. Each step then uses a bounded run-wide extraction pool followed by
serial per-lane merge and normalize continuations. A step barrier prevents
later consumers from starting until all earlier lanes are terminal and their
required normalized artifacts have crossed the typed handoff.

## Application Boundary

| Package | Implemented responsibility |
| --- | --- |
| `cmd/notarius` | Executable entry point and process exit delegation. |
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. |

## Core Packages

| Package | Implemented responsibility |
| --- | --- |
| `internal/core/artifacts` | Run-manifest and provenance models. |
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. |
| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. |
| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. |

## Framework Packages

| Package | Implemented responsibility |
| --- | --- |
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
| `internal/framework/debug` | Root-based framework and LLM debug recording. |

Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
serialized-validator, and
typed-validator interfaces. The runner owns handoff provenance, validation
sequencing, rejection handling, checkpoint and debug boundaries, and final
manifest assembly.

Artifact registries support heterogeneous typed extraction entries and
kind-specific merger, normalizer, and validator variants. Resolution derives a
lane's kind from its extractor, requires the matching codec, verifies exact Go
type equality across the lane, and records schema identity in the resolved lane
and pipeline digest. Registry entries carry separate option-validation and
run-local construction closures. Preparation injects shared dependencies and
constructs input, chunk, validators, ordered lanes, and output before source
parsing. Production modules use strict construction-time option decoding, and
LLM-backed modules retain the injected shared client. The D&D family registers
the canonical `dnd/spell-list`, `dnd/npc-list`, `dnd/combat-turn-list`,
`dnd/npc-interaction-list`, and `dnd/scene-description-list` codecs, typed
spell, NPC, combat, interaction, and scene-description extractors and
normalizers, validators,
plus kind-specific generic merge strategies; generic JSON validators use the
serialized-validation contract. The runner executes lanes through
private exact-type-checked closures, coordinates extract results independently
of completion timing, and serializes artifacts only through their codec at
checkpoint, debug, and output boundaries.

## Production Extensions

The canonical catalogs of user-selectable
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) keys are in
Configuration. The implemented module packages are:

| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| `internal/modules/dnd` | Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, combat-turn-list, combat-turn, NPC-interaction-list, and scene-description-list artifact types. |
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
| `internal/modules/dnd/codec/npcs` | Strictly decodes and stably encodes the durable D&D NPC-list representation. |
| `internal/modules/dnd/codec/combatturns` | Strictly decodes and stably encodes the durable D&D combat-turn-list representation. |
| `internal/modules/dnd/codec/npcinteractions` | Strictly decodes and stably encodes the durable D&D NPC-interaction-list representation. |
| `internal/modules/dnd/codec/scenedescriptions` | Strictly decodes and stably encodes the durable D&D scene-description-list representation. |
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
| `internal/modules/dnd/extract/npcs` | Maps private structured model output to canonical source-grounded D&D NPC lists. |
| `internal/modules/dnd/extract/combatturns` | Maps private structured model output to source-grounded D&D combat-turn candidates and preserves chronology and invalid candidate values for validators. |
| `internal/modules/dnd/extract/npcinteractions` | Maps private structured model output to current-source NPC interaction candidates grounded by a required registry. |
| `internal/modules/dnd/extract/scenedescriptions` | Maps one private scene description to the current accepted chunk's ID and exact range. |
| `internal/modules/dnd/npcinteractions` | Owns canonical source-reference ordering, occurrence ordering, valid-evidence checks, and exact interaction identity shared by normalization and invariant validation. |
| `internal/modules/dnd/normalize/combatturns` | Canonicalizes and orders merged combat turns, applies exact NPC identity matches, and collapses only exact valid-evidence duplicates. |
| `internal/modules/dnd/normalize/npcinteractions` | Canonicalizes required-registry names, orders interaction occurrences, and collapses only exact valid-evidence duplicates. |
| `internal/modules/dnd/normalize/scenedescriptions` | Trims, source-orders, and removes only exactly identical scene descriptions while rejecting ID and range conflicts. |
| `internal/modules/dnd/validate/combatturns` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for the production combat chains. |
| `internal/modules/dnd/validate/npcinteractions` | Provides deterministic shape, registry, source-reference, source-relatedness, and normalized-invariant validation for interaction chains. |
| `internal/modules/dnd/validate/scenedescriptions` | Provides deterministic shape, exact extraction attachment, source-relatedness, and normalized-invariant validation for scene-description chains. |
| `internal/modules/dnd/npcs/registry` | Resolves validated normalized NPC references into immutable grounding data and exact identity lookup. |
| `internal/modules/dnd/npcs/identity` | Owns Unicode-aware NPC identity, ID derivation, and registry collision validation. |
| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by canonical name, unions exact evidence, and reports bounded warnings. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |

`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
reference declarations, prompt input assembly, source-unit reference helpers,
and bounded diagnostics under `internal/modules/dnd/shared/diagnostics`.
The shared NPC grounding fragment is mounted for D&D prompts and is owned by
this package. Domain-neutral prompt filesystem composition lives in
`internal/framework/promptfs`.

The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
External references are strictly decoded and identity-validated during
preparation; generated references are decoded and identity-validated at the
ordered step handoff. Both paths retain canonical registry JSON for provenance
and emit a names-only projection to operation-time spell, combat, and
interaction prompts. Combat and interaction normalization use the canonical
registry for exact name lookup. The
framework records generated identity and bounded producer provenance, while
the raw external reference remains independently tracked by pipeline
provenance. An absent registry is represented only by the empty prompt value
`{"npcs":[]}`. Spell
and combat consumers use this shared boundary without changing their public
module contracts. Interaction consumers require it and retain only current
transcript references as durable evidence.

Generic validators under `internal/modules/generic/validate` provide
unconditional test decisions, JSON syntax validation, and JSON Schema
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
consume the canonical spell-list type directly to provide shape,
effective-catalog, source-reference, and source-relatedness decisions.

Production composition is grouped behind package-family registrars, and every
implemented production extension uses its domain-first tree:

| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |

The CLI allocates the framework registries and asset registry, then invokes
these registrars in generic, Seriatim, and D&D order.

Implementation details for all production extensions are in
[Module Internals](modules.md).

## Run-State Components

| Surface | Implemented owners | Internal purpose |
| --- | --- | --- |
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. |
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |

Physical layout, cleanup, recovery, and sensitive-data handling are defined
in [Operations](../operations.md). Concrete modules receive recorder
interfaces and request data, not physical state roots.

## Focused Documentation

- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
  checkpoint/debug hooks, and result assembly.
- [Module Internals](modules.md): production modules, validators, assets,
  registration, and the contributor recipe for adding an extension.
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
  assets, scheduling, profile recording, and redaction.
- [Run State Internals](state.md): output, cache, debug collaborator
  composition, and path safety.