Refocus developer and internal documentation
This commit is contained in:
@@ -1,127 +1,105 @@
|
||||
# Internal Overview
|
||||
|
||||
This document maps the implemented Notarius components and their ownership. It
|
||||
complements the durable invariants in [Architecture](../policy/architecture.md)
|
||||
and links to focused internal documentation for deeper behavior.
|
||||
This document inventories the implemented Notarius components. Normative
|
||||
boundaries and dependency direction belong in
|
||||
[Architecture](../policy/architecture.md); external behavior belongs in the
|
||||
[CLI](../cli.md), [Configuration](../config.md),
|
||||
[Operations](../operations.md), and [integration contracts](../integrations/).
|
||||
|
||||
## Execution Path
|
||||
|
||||
The executable delegates to the CLI, which resolves configuration and wires the
|
||||
production application around the framework runner:
|
||||
`cmd/notarius` delegates to `internal/cli`, the production composition root.
|
||||
The CLI loads configuration, builds the production catalogs and runtime
|
||||
collaborators, invokes `internal/framework/pipeline`, and places the logical
|
||||
output files returned by the runner. Diagnostics, checkpoints, and debug
|
||||
recorders are optional side-channel collaborators supplied at this boundary.
|
||||
|
||||
```text
|
||||
cmd/notarius
|
||||
-> internal/cli
|
||||
-> config resolution + production registries + LLM client
|
||||
-> input -> chunk -> extract -> merge -> normalize -> output
|
||||
-> durable output writes
|
||||
|
||||
Pipeline side channels:
|
||||
diagnostics checkpoints debug artifacts
|
||||
```
|
||||
|
||||
Pipeline execution is serial. Configuration selects modules for the fixed stage
|
||||
shape; registries construct them after profile, capability, validator, and
|
||||
reference resolution.
|
||||
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
|
||||
a sorted set of artifact lanes before the runner constructs any stage module.
|
||||
|
||||
## Application Boundary
|
||||
|
||||
`cmd/notarius` contains the executable entry point and delegates process exit
|
||||
behavior to `internal/cli`.
|
||||
|
||||
`internal/cli` owns command parsing, configuration discovery, production module
|
||||
and validator registration, prompt asset collection, production LLM client
|
||||
construction, reference preparation, workspace recorder setup, durable output
|
||||
writes, and user-facing stdout, stderr, and exit codes. It is the composition
|
||||
root for concrete production packages.
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
||||
| `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
||||
|
||||
## Core Packages
|
||||
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/core/artifacts` | Run manifests and artifact serialization shapes. |
|
||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline configuration. |
|
||||
| `internal/core/diagnostics` | Diagnostics run directories, artifact writers, atomic writes, and retention decisions. |
|
||||
| `internal/core/source` | Generic source documents, units, references, and validation. |
|
||||
| `internal/core/workspace` | Workspace settings, safe paths and writes, checkpoint identities, and checkpoint manifest types. |
|
||||
|
||||
These packages provide concrete, deterministic models and policy. Production
|
||||
module registration occurs at the CLI boundary rather than in core packages.
|
||||
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||
| `internal/core/source` | Generic source documents, units, references, lookup, and validation. |
|
||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
||||
|
||||
## Framework Packages
|
||||
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/framework/contracts` | Stage, validator, reference, output, and structured LLM interfaces and request/result types. |
|
||||
| `internal/framework/pipeline` | Module registries, profile resolution, capability checks, reference materialization, validation chains, retries, orchestration, warnings, and manifest population. |
|
||||
| `internal/framework/contracts` | Stage, validator, reference, output, and structured-completion interfaces and data types. |
|
||||
| `internal/framework/pipeline` | Registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
|
||||
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema asset registration, scheduling, profile recording, and secret redaction. |
|
||||
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload envelopes. |
|
||||
| `internal/framework/debug` | Workspace-backed framework and LLM debug artifacts. |
|
||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
||||
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
||||
|
||||
Framework contracts carry raw stage outputs between modules. The runner owns
|
||||
provenance, validation sequencing, rejection handling, checkpoint boundaries,
|
||||
debug boundaries, and final manifest assembly.
|
||||
Framework contracts carry raw stage results between implementations. The
|
||||
runner owns handoff provenance, validation sequencing, rejection handling,
|
||||
checkpoint and debug boundaries, and final manifest assembly.
|
||||
|
||||
## Production Modules
|
||||
## Production Extensions
|
||||
|
||||
Production implementations live under `internal/modules` and register through
|
||||
the CLI catalog.
|
||||
The canonical catalogs of user-selectable
|
||||
[module](../config.md#implemented-production-modules) and
|
||||
[validator](../config.md#implemented-production-validators) keys are in
|
||||
Configuration. The implemented module packages are:
|
||||
|
||||
| Stage | Module key | Package | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| Input | `seriatim` | `internal/modules/input/seriatim` | Converts Seriatim transcript JSON into the generic source model. |
|
||||
| Chunk | `generic` | `internal/modules/chunk/generic` | Splits ordered source units by configured unit counts and overlap. |
|
||||
| Chunk | `dnd/scenes` | `internal/modules/chunk/dnd/scenes` | Uses structured LLM output to create contiguous D&D scene chunks. |
|
||||
| Extract | `dnd/spells` | `internal/modules/extract/dnd/spells` | Extracts source-grounded D&D spell-cast artifacts. |
|
||||
| Merge | `appendorder` | `internal/modules/merge/appendorder` | Combines accepted extract outputs in chunk order. |
|
||||
| Normalize | `noop` | `internal/modules/normalize/noop` | Preserves accepted merged output unchanged. |
|
||||
| Output | `json` | `internal/modules/output/json` | Encodes manifests, indexes, warnings, rejections, and accepted lane payloads as logical JSON files. |
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/modules/input/seriatim` | Parses the supported Seriatim transcript format into the generic source model. |
|
||||
| `internal/modules/chunk/generic` | Splits ordered source units by unit count and overlap. |
|
||||
| `internal/modules/chunk/dnd/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
||||
| `internal/modules/extract/dnd/spells` | Produces source-grounded D&D spell-cast raw output. |
|
||||
| `internal/modules/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
||||
| `internal/modules/normalize/noop` | Preserves accepted merged output. |
|
||||
| `internal/modules/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
||||
|
||||
`internal/modules/sharedassets` composes shared prompt filesystems.
|
||||
`internal/modules/sharedassets/dnd` owns shared D&D prompt fragments, reference
|
||||
slots, prompt input assembly, and source-unit reference helpers.
|
||||
`internal/modules/sharedassets/dnd` owns reusable D&D prompt fragments,
|
||||
reference declarations, prompt input assembly, and source-unit reference
|
||||
helpers.
|
||||
|
||||
## Validators
|
||||
Concrete validators live under `internal/validators`. Generic packages provide
|
||||
unconditional test decisions, JSON syntax validation, and JSON Schema
|
||||
validation. D&D spell packages provide shape, source-reference, and
|
||||
source-relatedness decisions, with `spellpayload` holding their shared parser
|
||||
and lookup helpers. Production chain composition is owned by `internal/cli`.
|
||||
|
||||
Concrete validators live under `internal/validators` and register separately
|
||||
from stage modules. Generic validators cover unconditional test decisions, JSON
|
||||
syntax, and JSON Schema. D&D spell validators cover artifact shape, source
|
||||
reference validity, and source relatedness.
|
||||
Implementation details for all production extensions are in
|
||||
[Module Internals](modules.md).
|
||||
|
||||
The production default chain for `dnd/spells` extract output is registered
|
||||
centrally in `internal/cli`; module packages produce raw output but do not own
|
||||
the production approve/reject policy.
|
||||
## Run-State Components
|
||||
|
||||
## Files And Run State
|
||||
|
||||
Notarius keeps distinct output and inspection surfaces:
|
||||
|
||||
| Surface | Owner | Purpose |
|
||||
| Surface | Implemented owners | Internal purpose |
|
||||
| --- | --- | --- |
|
||||
| Durable output | Output module and CLI writer | User-consumable run files. |
|
||||
| Diagnostics | `internal/core/diagnostics` and CLI | Redacted run inspection, reports, warnings, and failures. |
|
||||
| Checkpoints | `internal/framework/checkpoint` | Validated stage reuse for explicit resume. |
|
||||
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Sensitive framework-boundary and LLM call inspection. |
|
||||
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
||||
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
|
||||
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
|
||||
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
|
||||
|
||||
Workspace settings determine whether and where diagnostics, checkpoints, and
|
||||
debug artifacts are written. Concrete stage modules do not receive workspace
|
||||
paths.
|
||||
Physical layout, retention, recovery, and sensitive-data handling are defined
|
||||
in [Operations](../operations.md). Concrete stage modules receive recorder
|
||||
interfaces and request data, not workspace paths.
|
||||
|
||||
## Focused Internal Documentation
|
||||
## Focused Documentation
|
||||
|
||||
- [Pipeline Internals](pipeline.md): resolution, execution, validators,
|
||||
references, retries, checkpoints, outputs, and manifests.
|
||||
- [Module Internals](modules.md): production module contracts, capabilities,
|
||||
options, prompts, schemas, and registration.
|
||||
- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
|
||||
checkpoint/debug hooks, and result assembly.
|
||||
- [Module Internals](modules.md): production modules, validators, assets,
|
||||
registration, and the contributor recipe for adding an extension.
|
||||
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
|
||||
assets, scheduling, profile recording, and redaction.
|
||||
- [Diagnostics Internals](diagnostics.md): diagnostics files, retention,
|
||||
failure behavior, and path safety.
|
||||
|
||||
## Test Surfaces
|
||||
|
||||
The repository uses focused package tests, registry and pipeline composition
|
||||
tests, a fake-backed walking skeleton, fixture-driven CLI coverage, and local
|
||||
test servers for LLM integration behavior. Tests do not require real provider
|
||||
calls.
|
||||
- [Diagnostics Internals](diagnostics.md): scoped writers, retention
|
||||
coordination, CLI failure flow, and path safety.
|
||||
|
||||
Reference in New Issue
Block a user