142 lines
8.5 KiB
Markdown
142 lines
8.5 KiB
Markdown
# Internal Overview
|
|
|
|
This document inventories the implemented Notarius components. Normative
|
|
boundaries and dependency direction belong in
|
|
[Architecture](../policy/architecture.md); external behavior belongs in the
|
|
[CLI](../cli.md), [Configuration](../config.md),
|
|
[Operations](../operations.md), and [integration contracts](../integrations/).
|
|
|
|
## Execution Path
|
|
|
|
`cmd/notarius` delegates to `internal/cli`, the production composition root.
|
|
The CLI loads configuration, builds the production catalogs and runtime
|
|
collaborators, invokes `internal/framework/pipeline`, and places the logical
|
|
output files returned by the runner. Diagnostics, checkpoints, and debug
|
|
recorders are optional side-channel collaborators supplied at this boundary.
|
|
|
|
Resolution produces a fixed ordered workflow and a sorted set of artifact
|
|
lanes. Preparation constructs the complete module and validator set before the
|
|
runner receives source bytes. Source parsing and chunking are serial; extraction
|
|
uses a bounded run-wide worker pool, followed by serial per-lane merge and
|
|
normalize continuations that may overlap across lanes.
|
|
|
|
## Application Boundary
|
|
|
|
| Package | Implemented responsibility |
|
|
| --- | --- |
|
|
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
|
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
|
|
|
## Core Packages
|
|
|
|
| Package | Implemented responsibility |
|
|
| --- | --- |
|
|
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
|
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
|
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
|
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
|
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
|
|
|
## Framework Packages
|
|
|
|
| Package | Implemented responsibility |
|
|
| --- | --- |
|
|
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
|
|
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
|
|
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
|
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
|
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
|
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
|
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
|
|
|
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
|
serialized-validator, and
|
|
typed-validator interfaces. The runner owns handoff provenance, validation
|
|
sequencing, rejection handling, checkpoint and debug boundaries, and final
|
|
manifest assembly.
|
|
|
|
Artifact registries support heterogeneous typed extraction entries and
|
|
kind-specific merger, normalizer, and validator variants. Resolution derives a
|
|
lane's kind from its extractor, requires the matching codec, verifies exact Go
|
|
type equality across the lane, and records schema identity in the resolved lane
|
|
and pipeline digest. Registry entries carry separate option-validation and
|
|
run-local construction closures. Preparation injects shared dependencies and
|
|
constructs input, chunk, validators, ordered lanes, and output before source
|
|
parsing. Production modules use strict construction-time option decoding, and
|
|
LLM-backed modules retain the injected shared client. The D&D family registers
|
|
the canonical `dnd/spell-list` codec, typed spell extractor and validators, and
|
|
kind-specific generic merge and normalize strategies; generic JSON validators
|
|
use the serialized-validation contract. The runner executes lanes through
|
|
private exact-type-checked closures, coordinates extract results independently
|
|
of completion timing, and serializes artifacts only through their codec at
|
|
checkpoint, debug, and output boundaries.
|
|
|
|
## Production Extensions
|
|
|
|
The canonical catalogs of user-selectable
|
|
[module](../config.md#implemented-production-modules) and
|
|
[validator](../config.md#implemented-production-validators) keys are in
|
|
Configuration. The implemented module packages are:
|
|
|
|
| Package | Implemented responsibility |
|
|
| --- | --- |
|
|
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
|
|
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
|
|
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
|
| `internal/modules/dnd` | Owns the canonical D&D spell-list and spell-cast artifact types. |
|
|
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
|
|
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
|
|
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
|
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
|
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
|
|
|
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
|
reference declarations, prompt input assembly, and source-unit reference
|
|
helpers. Domain-neutral prompt filesystem composition lives in
|
|
`internal/framework/promptfs`.
|
|
|
|
Generic validators under `internal/modules/generic/validate` provide
|
|
unconditional test decisions, JSON syntax validation, and JSON Schema
|
|
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
|
|
consume the canonical spell-list type directly to provide shape,
|
|
source-reference, and source-relatedness decisions.
|
|
|
|
Production composition is grouped behind package-family registrars, and every
|
|
implemented production extension uses its domain-first tree:
|
|
|
|
| Package | Implemented responsibility |
|
|
| --- | --- |
|
|
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
|
|
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
|
|
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
|
|
|
|
The CLI allocates the framework registries and asset registry, then invokes
|
|
these registrars in generic, Seriatim, and D&D order.
|
|
|
|
Implementation details for all production extensions are in
|
|
[Module Internals](modules.md).
|
|
|
|
## Run-State Components
|
|
|
|
| Surface | Implemented owners | Internal purpose |
|
|
| --- | --- | --- |
|
|
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
|
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
|
|
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
|
|
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
|
|
|
|
Physical layout, retention, recovery, and sensitive-data handling are defined
|
|
in [Operations](../operations.md). Concrete stage modules receive recorder
|
|
interfaces and request data, not workspace paths.
|
|
|
|
## Focused Documentation
|
|
|
|
- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
|
|
checkpoint/debug hooks, and result assembly.
|
|
- [Module Internals](modules.md): production modules, validators, assets,
|
|
registration, and the contributor recipe for adding an extension.
|
|
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
|
|
assets, scheduling, profile recording, and redaction.
|
|
- [Diagnostics Internals](diagnostics.md): scoped writers, retention
|
|
coordination, CLI failure flow, and path safety.
|