eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refocus developer and internal documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-17 03:06:23 +00:00
parent b1fe9dc5a7
commit 6e6375521d
7 changed files with 549 additions and 785 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
docs/development.md
View File

@@ -4,41 +4,16 @@ This is the first-read landing page for people and LLM coding agents working on
Notarius. It provides a concise repository orientation and routes each kind of
change to its canonical documentation.
## Orientation
Notarius is a small Go application for extracting structured data from source
material. It is a general extraction platform with an initial Seriatim and D&D
implementation. Configured modules run through a fixed workflow:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
The CLI is the application boundary. Core packages own deterministic models and
policy, framework packages own reusable contracts and orchestration, and module
and validator packages own concrete behavior.
## Repository Map
- `cmd/notarius`: executable entry point.
- `internal/cli`: CLI behavior and production composition.
- `internal/core`: deterministic source, config, artifact, diagnostics, and
workspace packages.
- `internal/framework`: contracts, pipeline orchestration, validation helpers,
checkpoints, debug recording, and LLM runtime plumbing.
- `internal/modules`: concrete implementations of the six pipeline stages.
- `internal/validators`: concrete output validators.
- `docs`: canonical policy, reference, integration, internal, ADR, and roadmap
documentation.
- `examples`: maintained, secret-free example inputs and configuration.
See [Internal Overview](internal/overview.md) for the implemented component
map and links to focused internal documentation.
Notarius is a Go CLI for configured structured extraction workflows. Start with
the [README](../README.md) for product context, [Architecture](policy/architecture.md)
for system boundaries, and [Internal Overview](internal/overview.md) for the
implemented component map.
## What To Read
| When working on | Read | Why |
| --- | --- | --- |
| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internals. |
| Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
@@ -63,13 +38,3 @@ go test ./...
go vet ./...
go build ./cmd/notarius
```
## Universal Reminders
- Keep non-roadmap documentation limited to implemented behavior.
- Update affected documentation and maintained examples in the same change as
behavior.
- Use fakes, fixtures, or local test servers instead of real external services
in tests.
- Do not expose secrets in code, errors, logs, diagnostics, manifests,
documentation, or examples.

151
docs/internal/diagnostics.md
View File

@@ -1,102 +1,93 @@
# Diagnostics Internals
Diagnostics internals live in `internal/core/diagnostics`. Operator-facing run
behavior is documented in [Operations](../operations.md).
## Purpose
Diagnostics provide local inspection artifacts for a run without becoming the
durable output contract. Durable user output is produced by output modules and
written by the CLI.
Diagnostics must not expose secrets.
`internal/core/diagnostics` provides the scoped writer and retention decision
used by `internal/cli`. The physical layout, artifact inventory, retention
semantics, failure inspection, and cleanup procedures are canonical in
[Operations](../operations.md#diagnostics-directory). Configuration fields and
defaults are canonical in [Configuration](../config.md#diagnostics).
## Run Directory
`NewRunDirectory(workDir, retention)` creates:
`NewRunDirectory` normalizes empty constructor inputs, creates the effective
diagnostics root when needed, and allocates a unique timestamp-based child
directory. It retries a bounded number of collisions before failing. The
resulting `RunDirectory` retains its creation time and retention mode for later
metadata and cleanup decisions.
```text
<workDir>/run-<unix-nanoseconds>/
```
The package does not resolve workspace configuration. `internal/cli` derives
effective workspace settings first and passes the diagnostics root into the
constructor.
If `workDir` is empty, it defaults to `/tmp/notarius`. Empty retention defaults
to `auto`.
## Scoped Writers
The CLI passes the effective diagnostics root from workspace configuration.
When `workspace.directory` is set and diagnostics are enabled, that root is
`<workspace.directory>/diagnostics`. The legacy diagnostics work directory and
`--diagnostics-dir` still pass a diagnostics-only root to this constructor.
Typed methods on `RunDirectory` write invocation metadata, redacted effective
configuration, resolved pipeline/reference data, checkpoint events, source data
when explicitly requested, manifests, reports, warnings, and error text. The
current filenames and their operator-facing contents are listed in
[Operations](../operations.md#diagnostics-directory).
The writer makes the work directory if needed, then attempts to create a unique
run directory. It retries run ID creation a bounded number of times if a
collision occurs.
JSON methods indent their payload and append a newline. All artifact writes use
a temporary file in the target directory, apply the requested permissions, and
rename it into place. Artifact resolution accepts only a single relative base
name; absolute paths, separators, and paths escaping the run directory fail
before writing.
## Artifact Writers
## Redacted Configuration
Implemented artifact names:
`WriteRedactedEffectiveConfig` accepts a `RedactedDiagnosticsPayload` provider
rather than a raw config value. `internal/core/config` implements that contract
by cloning effective config data and removing secret-shaped values before JSON
encoding. The diagnostics package therefore never needs configuration-specific
field knowledge.
- `invocation.json`
- `effective-config.json`
- `resolved-pipeline.json`
- `resolved-references.json`
- `checkpoint-events.json`
- `source-document.json`
- `run-manifest.json`
- `run-report.json`
- `warnings.json`
- `error.log`
## Retention Coordination
JSON artifacts are encoded with indentation and a trailing newline. Writes are
atomic through a temporary file in the target directory followed by rename.
`ShouldRetainRunDirectory` is a pure decision over the effective retention mode,
run success, and warning presence. `ApplyRetention` uses that result to remove
only its own run directory. Unsupported modes retain data as a fail-safe, though
normal CLI execution rejects them during config validation.
Artifact names must be single relative file names. Absolute paths, path
separators, and names resolving outside the run directory are rejected.
The meaning of each supported mode belongs in
[Operations](../operations.md#retention); this package implements that contract
without loading config or inspecting run artifacts.
## Redacted Effective Config
## CLI State Flow
Diagnostics writers accept payloads that implement
`RedactedDiagnosticsPayload`. `internal/core/config` uses this to redact API
keys in effective config diagnostics while preserving resolved pipeline context.
When diagnostics are enabled, `internal/cli` creates the run directory after
configuration loading and before pipeline resolution. It then writes artifacts
as state becomes available: invocation data, effective resolution data,
pipeline results, and the final report. This ordering permits later failures to
retain the context already established.
The redaction path clones config data before replacing secret values.
Failures before construction have no `RunDirectory`. Later failures write an
error log, preserve any available partial manifest, and apply a failed-run
retention decision. A diagnostics write failure is itself a command failure so
the CLI does not report success after losing requested inspection data.
## Retention
When diagnostics are disabled, the CLI carries a nil run directory and the
shared `writeDiagnostics` helper turns writes into no-ops. User-facing errors
still go to stderr; that invocation behavior is documented in
[Operations](../operations.md#failures).
Retention is decided by `ShouldRetainRunDirectory`.
## Package Guarantees
- Failed runs are always retained.
- `always` retains successful runs.
- `never` removes successful runs.
- `auto` retains successful runs only when warnings exist.
- Unknown retention values are treated as retain by the retention decision, but
config validation rejects unsupported values before normal runs.
- A `RunDirectory` writes and removes only within its allocated directory.
- JSON and error artifacts use atomic replacement.
- Nil receivers and invalid typed payloads return errors rather than panicking.
- Retention never removes a failed run and never targets the diagnostics root.
- Diagnostics models contain inspection metadata, not the durable output
contract.
- Checkpoint and debug serializers remain separate framework components.
- Secret-handling follows the invariant in
[Architecture](../policy/architecture.md#state-output-and-safety).
`ApplyRetention` removes only the specific run directory.
## Tests To Inspect
## CLI Failure Behavior
When diagnostics are enabled, the CLI creates the diagnostics run directory
after config loading and before pipeline resolution. Failures before that point
do not have diagnostics.
When workspace diagnostics are explicitly disabled, the CLI does not create a
diagnostics run directory and skips diagnostics artifact writes. Failures are
still printed to stderr.
After diagnostics creation, run failures call `WriteErrorLog` and apply
retention with `RunSucceeded: false`, so the run directory remains available.
When the pipeline returns a partial manifest on failure, the CLI writes that
manifest before logging the failure.
## Invariants
- Diagnostics paths must be narrow and run-directory scoped.
- Writes should be atomic where practical.
- Secrets must be redacted.
- Diagnostics write failures are command failures because they can hide the
information needed for recovery.
- Durable output file contracts belong to output modules and integration docs,
not to diagnostics.
- Checkpoint and debug workspace files are separate framework-owned artifacts,
not diagnostics artifacts.
- `internal/core/diagnostics/run_dir_test.go`: allocation, artifact confinement,
atomic writes, retention, and failure behavior.
- `internal/core/diagnostics/artifacts_test.go`: stable artifact identifiers.
- `internal/core/config/redaction_test.go`: clone-and-redact payload behavior.
- `internal/core/workspace/settings_test.go`: effective diagnostics-root and
enablement handoff.
- `internal/cli/run_test.go`: creation timing, artifact sequencing, disabled
diagnostics, overrides, failures, and retention integration.

192
docs/internal/llm.md
View File

@@ -1,114 +1,132 @@
# LLM Runtime
# LLM Runtime Internals
The implemented LLM runtime lives in `internal/framework/llm`. It provides
transport-neutral structured completion contracts, a Scriptorium-backed
production client, concurrency scheduling, prompt/schema asset registration,
schema registry helpers, and secret redaction.
`internal/framework/llm` implements Notarius's transport boundary for structured
completion. It contains the Scriptorium adapter, concurrency scheduler,
prompt/schema registries, selected-profile recording, and provider-error
redaction.
## Contract
Provider-neutral ownership rules are defined in
[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
credentials, and concurrency settings are defined in
[Configuration](../config.md).
Modules depend on `contracts.StructuredLLMClient`:
## Structured Contract
```go
CompleteStructured(ctx, request, out) (response, error)
```
Modules and LLM-backed validators depend on
`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a
prompt and optional profile/session, supplies named input materials and
variables, and provides a caller-owned decoding target. A successful response
contains the validated raw structured bytes plus non-secret provider, model,
profile, and token metadata.
The request contains prompt ID/version, profile ID, session ID, prompt input
materials, and variables. The caller supplies a pointer target for decoded
structured output. The response also carries the raw structured output bytes
returned by the runtime so modules can preserve raw payloads in pipeline stage
outputs.
The caller owns prompt selection, response-schema selection, and interpretation
of the decoded result. `LLMInputMaterial` keeps source and reference bytes with
their origin metadata so the adapter can pass named artifacts to Scriptorium
without exposing Scriptorium types through stage contracts.
Modules that call the LLM own their prompts, schemas, prompt IDs, and
domain-specific interpretation. Validator packages own approve/reject policy,
and central catalog mappings decide which validators run by default. Provider
adapters should not contain domain-specific prompt logic.
## Production Construction
Prompt input materials carry source or reference bytes with optional origin
metadata. The Scriptorium-backed runtime receives them as named artifacts rather
than rendered prompt strings owned by Notarius modules.
`internal/cli` constructs the production runtime by:
## Production Client Construction
1. collecting embedded prompt and response-schema assets from production module
packages;
2. creating a `ScriptoriumClient` from the effective profile source;
3. attaching an `LLMProfileRecorder`;
4. creating a scheduler from the effective concurrency limit;
5. returning a `ScheduledClient` wrapper.
`internal/cli` builds the production LLM client from the effective config:
1. collect production Scriptorium prompt and schema assets from module packages;
2. create a Scriptorium-backed structured client using effective Scriptorium
profile source settings from `scriptorium.profile_dir` or
`scriptorium.profile_file`;
3. create a scheduler from global LLM concurrency;
4. wrap the client with `NewScheduledClient`;
5. let the runtime report non-secret profile manifest metadata after calls.
The runtime records the actual selected Scriptorium profile, provider, and model
used during execution. Manifest population does not rely on a precomputed
profile ID before pipeline execution.
Explicit profile validation applies to LLM-capable pipeline stages: chunk,
extract, merge, normalize, and LLM-backed validators with explicit
`llm_profile` values. Input, output, and deterministic validators do not call
the LLM. The `--llm-profile` run flag overrides effective chunk, extract, merge,
and normalize bindings; it does not override validator-specific profiles.
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
and validator bindings. It prepares a small internal check prompt for each ID so
missing or invalid profiles fail before pipeline execution. The runtime profile
override syntax and scope are defined in the
[CLI reference](../cli.md#run); binding rules are defined in
[Configuration](../config.md#module-bindings).
## Scriptorium Adapter
`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting
Notarius prompt requests into Scriptorium `RunRequest` values. It:
`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
It validates the decoding target and prompt identity, maps named input materials
to inline artifacts, forwards explicit profile and session context, delegates
rendering/provider execution/structured validation, and unmarshals successful
JSON into the caller target.
- validates the caller output target and prompt ID;
- converts `LLMInputMaterial` values into inline Scriptorium artifacts, using a
single space for empty material so optional blank references remain explicit;
- passes `session_id` through Scriptorium variables and request metadata when
present;
- sends explicit profile IDs only when the request supplies one;
- lets Scriptorium render prompts, call the configured provider, and validate
structured output;
- unmarshals successful JSON into the caller-provided target;
- returns the validated raw structured output bytes to the caller;
- maps token usage and selected profile/model metadata into the Notarius
response and manifest profile recorder.
Empty optional input material is represented by a single space so Scriptorium
retains the named input. The client returns Scriptorium's validated structured
bytes rather than re-encoding the caller target, allowing modules to preserve
the runtime result exactly.
Generated-output validation failures are returned as Notarius errors. Provider
and runtime errors are wrapped with prompt context and bearer tokens are
redacted from error strings. Prompt text, raw source input, reference content,
schema JSON, API keys, and bearer tokens are not added to default diagnostics or
run manifests.
Selected profile, provider, model, and token metadata are mapped into the
Notarius response. The recorder deduplicates profiles by identity and supplies
manifest-safe profile summaries after actual calls; manifest population does
not guess the selected prompt default in advance.
## Scheduler
Generated-output validation failures and provider failures are wrapped with
prompt context. Error strings pass through bearer-token redaction before they
cross the runtime boundary.
`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a
FIFO queue of waiters. Cancellation removes queued waiters or releases granted
permits.
## Scheduling
`NewScheduledClient` wraps any structured LLM client and runs each completion
inside the scheduler.
`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
acquisition increments the in-flight count; queued acquisition waits for a
permit or context cancellation. Cancellation removes a queued waiter, while a
cancelled waiter that has already received a permit releases it.
Effective concurrency is:
`ScheduledClient` acquires a permit around each structured completion and
defers release on every result path. The effective limit and default are
configuration facts in [Configuration](../config.md#defaults).
1. `concurrency.total_llm`, when greater than zero;
2. `1`.
## Prompt And Schema Assets
## Schema Registry
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
and rejects invalid or conflicting registrations. Production module packages
register their own prompt and schema assets; generic framework code contains no
D&D prompt content.
The framework schema registry embeds generic test schemas. It also exposes
helpers for caller-owned schemas:
Schema helpers load embedded JSON Schema with identity and digest metadata,
return defensive copies, and expose a diagnostics map that omits schema bytes.
The small framework registry contains only generic test schemas; production
schemas remain package-owned.
- `LoadResponseSchema`
- `LookupResponseSchema`
- `MustLookupResponseSchema`
- `ResponseSchema.DiagnosticsMap`
## Debug And Redaction Boundaries
`DiagnosticsMap` omits raw schema content and includes metadata such as key,
ID, version, name, and SHA-256.
The pipeline may wrap the client with a debug recorder that captures prepared
prompt/response material for an explicitly enabled debug run. Default
diagnostics and manifests receive identities, hashes, usage, and selected
profile summaries rather than prompt, source, reference, schema, or response
content.
Production modules own and register their Scriptorium prompt and schema assets.
Framework packages may collect those files but must not contain D&D-specific
prompt content.
The Scriptorium error wrapper removes bearer credential values from surfaced
provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
secret values elsewhere in the runtime. Config diagnostics use a separate
clone-and-redact path in `internal/core/config`. These mechanisms implement the
security invariant in
[Architecture](../policy/architecture.md#state-output-and-safety); operator
handling of debug data is defined in [Operations](../operations.md#debug).
## Secret Redaction
## Failure Behavior
Provider errors are redacted before surfacing through the Scriptorium-backed
client. Config diagnostics use redacted effective config payloads.
- Invalid targets, missing prompt IDs, malformed structured output, and
Scriptorium failures return contextual errors to the calling module.
- Scheduler construction rejects non-positive limits; acquisition respects
context cancellation.
- Asset registration rejects invalid roots, missing content, and path conflicts.
- Schema loading distinguishes missing assets, invalid JSON, and invalid
metadata.
- Profile validation errors occur during CLI preparation when an explicit
selected ID cannot be prepared.
Do not add raw provider request bodies, response bodies, API keys, or prompt
payloads to diagnostics by default.
## Tests To Inspect
- `internal/framework/llm/scriptorium_client_test.go` and
`scriptorium_api_test.go`: adapter mapping and local HTTP integration.
- `internal/framework/llm/scheduler_test.go` and
`scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
release.
- `internal/framework/llm/asset_registry_test.go` and
`schema_registry_test.go`: asset composition, validation, and defensive
copies.
- `internal/framework/llm/secrets_test.go`: provider-error redaction.
- `internal/cli/run_test.go`: profile validation, production client wiring,
manifest recording, and debug integration.
- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
asset registration.

368
docs/internal/modules.md
View File

@@ -1,268 +1,194 @@
# Modules
Production modules live under `internal/modules`. Each module implements one
contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
registers itself with the matching pipeline registry.
# Module And Validator Internals
The CLI production catalog currently registers only the modules listed here.
Validator implementations live under `internal/validators` and are registered
separately from modules. Production default validator chains are central CLI
catalog policy; module packages do not own their default validation chains.
Production stage implementations live under `internal/modules`; production
validators live under `internal/validators`. The selectable keys, configuration
options, reference slots, and default validator chain are canonical in the
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) catalogs in
Configuration.
## Contract Pattern
## Extension Pattern
A production module package should provide:
A stage module package provides a stable key, constructor, contract
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
tests. A validator package follows the same pattern with `ValidatorSpec` and the
validator registry.
- a stable module key;
- a constructor such as `New`;
- the relevant contract implementation;
- `ModuleSpec`;
- `Register`;
- focused tests for registration, options, contract behavior, and errors.
Specs expose capability and execution metadata without constructing an
implementation. Chunk, extract, merge, and normalize modules that accept
auxiliary material declare identical reference slots from both
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
that agreement. Runtime delivery uses the corresponding stage request's
`References` field.
Module specs should describe capabilities accurately. Resolution uses specs to
reject incompatible pipelines before execution.
LLM-backed extensions own their prompt definitions and response schemas under
package-local embedded assets. Shared filesystem composition belongs in
`internal/modules/sharedassets`; reusable D&D prompt fragments, reference
declarations, prompt-input assembly, and source-unit helpers belong in
`internal/modules/sharedassets/dnd`. Stage contracts expose only Notarius
structured-completion types, not Scriptorium public types.
Chunk, extract, merge, and normalize modules that accept auxiliary reference material
must declare slots through both `ReferenceSlots()` and
`ModuleSpec().ReferenceSlots`. The runtime slot list and registry metadata
should match so config validation can inspect slots without constructing module
instances. A slot declaration names the slot, whether it is required, accepted
media types, whether multiple items are allowed, and any byte limit. Empty
`AcceptedMediaTypes` means any inferred media type is accepted, though the file
must still be UTF-8 text. When a slot declares accepted media types, Notarius
compares the canonical base media type inferred from the file extension,
case-insensitively and without parameters.
Reference material may inform a module or prompt but must not become source
evidence. The resolver and materializer behavior is described in
[Pipeline Internals](pipeline.md#reference-materialization).
The resolver materializes reference content for chunk, extractor, merger, and
normalizer targets. Runtime delivery uses `contracts.ChunkRequest.References`,
`contracts.ExtractionRequest.References`, `contracts.MergeRequest.References`,
and `contracts.NormalizeRequest.References`. Reference material is not source
evidence and must not be converted into `SourceRef` values. If a module prompt
uses references, pass them as prompt input materials through the structured LLM
request. Prompt metadata hashes remain based on prompt asset source, not
rendered reference bytes.
## Input Adapter
LLM-backed modules own Scriptorium prompt definitions and response schemas in
their embedded assets. Module-owned prompts live under each module's shallow
`assets/prompts` tree and schemas live under `assets/schemas`. Generic shared
prompt filesystem composition lives under `internal/modules/sharedassets`.
Common D&D prompt fragments, reference slot helpers, prompt input assembly, and
reference rendering live under `internal/modules/sharedassets/dnd`. Module
contracts should expose prompt IDs, versions, input material names, and
non-secret prompt/schema hashes through manifest metadata; they should not
expose Scriptorium public types through chunk, extract, merge, or normalize contracts.
### `internal/modules/input/seriatim`
Chunk modules receive the structured LLM client, configured Scriptorium profile
ID, prompt session ID, and raw source input material through
`contracts.ChunkRequest` when they need model-backed chunking. The pipeline
runner validates generic chunk result invariants before extraction; module-owned
policies may be stricter but must stay within the module package.
The adapter decodes the supported transcript JSON, selects the source identity,
computes the raw-input digest, validates segments, and maps each segment into a
generic source unit with speaker and timestamp metadata. Its spec advertises the
transcript capabilities consumed by D&D modules.
Normalize modules receive the structured LLM client, configured Scriptorium
profile ID, prompt session ID, and reference material through
`contracts.NormalizeRequest` when they need model-backed reconciliation.
Parsing is strict about required values and duplicate unit IDs but deliberately
ignores unrelated Seriatim fields. The external format and derived-identity
rules are defined in the
[Seriatim contract](../integrations/seriatim.md).
Merge modules receive the structured LLM client, configured Scriptorium profile
ID, prompt session ID, raw source input material, and reference material through
`contracts.MergeRequest` when they need model-backed merge behavior.
## Chunkers
## `seriatim` Input
### `internal/modules/chunk/generic`
Package: `internal/modules/input/seriatim`
The generic chunker validates the source document, walks units in configured
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
Overlap changes the next window start but never reorders units. It records the
first and last unit and unit count in chunk metadata.
The `seriatim` adapter parses Seriatim transcript JSON into a generic source
document. It owns transcript JSON details, source ID selection, source digest
creation, transcript segment validation, and segment metadata mapping.
The accepted options and defaults are defined in
[Configuration](../config.md#implemented-production-modules). Generic
framework validation canonicalizes the returned unit slices before extraction.
Provides:
### `internal/modules/chunk/dnd/scenes`
- `source.transcript`
- `transcript.speaker`
- `transcript.timestamps`
The scene chunker prepares a structured Scriptorium request from the full
transcript, session, and optional D&D reference inputs. It validates the model's
scene boundaries against source-unit IDs and converts them into deterministic
chunks.
External JSON shape belongs in the Seriatim integration doc.
Scene validation requires sequential, contiguous, non-overlapping coverage from
the first source unit through the last. Each chunk contains JSON scene content
and module-owned metadata for the scene description, boundaries, confidence,
participants, and unit count. Boundary caveats become warnings. Malformed
structured output is returned as an error; there is no fallback chunker.
## `generic` Chunker
The package embeds its prompt and response schema and reports their non-secret
identity and hashes through singleton module metadata. Shared D&D assets supply
reference declarations and prompt inputs; their user-facing keys and accepted
file types remain canonical in [Configuration](../config.md).
Package: `internal/modules/chunk/generic`
## Extractor
The `generic` chunker splits source units into ordered chunks. It validates the
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
and records chunk metadata for start unit, end unit, and unit count.
### `internal/modules/extract/dnd/spells`
The pipeline runner canonicalizes chunk units from the source document by
integer ID before extractors and mergers run. Chunkers also populate chunk
start and end unit IDs, content bytes, and media type. Chunker-owned context
should stay in `SourceChunk.Metadata`.
The spell extractor prepares a structured request from one chunk, the
chunk-scoped source input, the session, and optional D&D reference inputs. It
decodes the model response, assigns the generic source identity to every source
reference, canonicalizes duplicate references, orders spell casts by their
earliest cited unit, and returns raw JSON plus response-schema provenance.
Options:
The package owns its embedded prompt, response schemas, and prompt/schema
manifest metadata. Shared D&D helpers keep prompt input names and source-unit
reference conversion consistent with the scene chunker. The extractor produces
raw output; production validators own approval policy.
- `max_units`: positive integer, default `50`;
- `overlap_units`: non-negative integer, default `0`, and less than
`max_units`.
The durable payload and manifest metadata shapes are defined in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
Provides:
## Merger And Normalizer
- `chunks`
### `internal/modules/merge/appendorder`
## `dnd/scenes` Chunker
The merger preserves extract-result order. It passes through one JSON result,
concatenates a common top-level array field across multiple JSON objects, and
otherwise emits an array of the decoded values. It rejects invalid JSON and
non-JSON media types, and it preserves compatible schema provenance.
Package: `internal/modules/chunk/dnd/scenes`
### `internal/modules/normalize/noop`
The `dnd/scenes` chunker uses the structured LLM client to divide transcript
source units into coherent D&D scenes. It supplies the embedded Scriptorium
prompt ID, prompt version, transcript input material, response schema, and
session ID to the runtime; validates model-authored source-unit boundaries; and
converts each scene into a deterministic source chunk.
The normalizer defensively clones the accepted merge result, including payload
bytes, metadata, warnings, and schema provenance, without changing its logical
content.
Its prompt definition lives under `assets/prompts` and its schema under
`assets/schemas`. Shared reusable D&D prompt fragments are provided by
`internal/modules/sharedassets/dnd` and referenced from prompt definitions under
`./sharedassets/`.
## Output Encoder
Requires:
### `internal/modules/output/json`
- `source.transcript`
The JSON encoder sorts normalized results by lane, derives collision-checked
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
media types, unsafe names, and sanitized-name collisions are errors.
Provides:
The encoder returns logical files only. The CLI places them on disk, and the
[JSON output contract](../integrations/json-output.md) defines their external
paths and schemas.
- `chunks`
- `chunks.scenes`
## Generic Validators
Options: none. Non-empty options are rejected.
The unconditional accept and reject validators provide deterministic production
registrations used primarily for controlled composition and tests.
The chunker enforces full source-unit coverage from the first source unit to the
last, sequential contiguous scenes, and no overlap. Its LLM-facing schema uses
integer `start_unit_id` and `end_unit_id` values matching source-unit IDs. It
assigns chunk IDs such as `scene-000001`, emits JSON chunk content, and stores
scene metadata including title, primary mode, participants, summary, boundary
note, confidence, boundary unit IDs, and unit count. Boundary caveats become
warnings with reason code
`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed
structured output rather than silently dropped.
The JSON syntax validator uses `encoding/json` to reject malformed payloads. The
JSON Schema validator requires schema bytes on the validation request, parses
the instance and schema with `jsonschema`, and distinguishes payload rejection
from schema loading or compilation errors. Neither validator calls the LLM.
Malformed model output fails explicitly rather than falling back to another
chunker. The chunker exposes prompt and response-schema provenance through
top-level `module_metadata.chunker` without raw prompts, raw schemas, source
text, or secrets.
## D&D Spell Validators
## `dnd/spells` Extractor
`internal/validators/extract/dnd/spells/spellpayload` provides strict decoding,
shape checks, source-reference candidates, and cited-text lookup shared by the
three validators.
Package: `internal/modules/extract/dnd/spells`
The shape validator rejects malformed JSON, unknown fields, missing or empty
spell fields, and empty reference lists. The source-reference validator applies
generic source-reference validation to every cited range. The relatedness
validator approves structurally valid payloads but warns when a case-insensitive
spell name is absent from all cited source text. It leaves malformed payloads to
the earlier validators in the configured chain.
The `dnd/spells` extractor owns D&D spell-cast extraction semantics. It
supplies the embedded Scriptorium prompt ID, prompt version, chunk-scoped
transcript input material, reference input materials, response schema, and
session ID to the runtime; then returns the structured LLM `spell_casts`
response as raw JSON.
Its LLM-facing source-reference schema uses integer `start_unit_id` and
`end_unit_id` values matching source-unit IDs.
Its prompt definition lives under `assets/prompts` and its schema under
`assets/schemas`. Shared reusable D&D prompt fragments are provided by
`internal/modules/sharedassets/dnd` and referenced from prompt definitions under
`./sharedassets/`.
Requires:
- `chunks`
- `source.transcript`
Provides:
- `dnd.spell_casts`
Response schema identity:
- schema ID: `notarius.dnd.spells`
- schema name: `notarius_dnd_spells_v1`
- schema version: `v1`
The extractor adds prompt and response-schema provenance to lane manifest
metadata under `artifact_lanes[].metadata.extractor`. Durable raw output
details belong in the
[D&D spell raw output contract](../integrations/dnd-spell-artifacts.md).
The production catalog validates `dnd/spells` raw extract output with generic
JSON validators followed by D&D spell validators under
`internal/validators/extract/dnd/spells`. The extractor itself remains
responsible for prompt, schema, and raw output production rather than
approve/reject policy.
The `dnd/scenes` chunker and `dnd/spells` extractor declare optional `players`,
`party`, and `glossary` reference slots accepting UTF-8 plain text, Markdown,
YAML, or JSON. They also accept `roster` as a deprecated compatibility alias for
`party`. Their prompts frame references as supporting disambiguation material
only; spell-cast artifacts must still be grounded in the source transcript.
## `appendorder` Merger
Package: `internal/modules/merge/appendorder`
The `appendorder` merger preserves chunk order for raw extract outputs. A
single JSON extract output is passed through as the merge output. Multiple JSON
object outputs with one common top-level array field are merged by concatenating
that array field in chunk order. Other valid JSON shapes are merged as a JSON
array of decoded values in chunk order. Non-JSON media types and invalid JSON
are rejected.
Provides:
- `merged`
## `noop` Normalizer
Package: `internal/modules/normalize/noop`
The `noop` normalizer clones the raw merge output and returns it unchanged.
Requires:
- `merged`
Provides:
- `normalized`
## `json` Output
Package: `internal/modules/output/json`
The `json` output encoder converts normalized raw outputs, rejected raw outputs,
warnings, and the run manifest into logical JSON output files. It writes one
payload file per lane under `lanes/` and sanitizes lane IDs for file names.
Normalized output payloads must be valid `application/json`.
Requires:
- `normalized`
Provides:
- `encoded`
Durable output file shapes belong in the
[JSON output contract](../integrations/json-output.md). Operator behavior
belongs in [Operations](../operations.md).
These validators are deterministic. Their selectable keys and production order
are defined in
[Configuration](../config.md#implemented-production-validators); their durable
payload rules are defined in the
[artifact contract](../integrations/dnd-spell-artifacts.md).
## Production Registration
Production registration is centralized in `internal/cli/catalog.go`.
`internal/cli/catalog.go` builds the production registries, registers module and
validator constructors, installs default validator-chain mappings, and exposes
the matching catalog for resolution. It also collects prompt assets from
LLM-backed packages before constructing the production client.
Do not make framework code import production modules. The CLI wires production
modules at the application boundary; tests may provide fake registries or fake
catalogs directly.
Framework packages must not import production extensions. Tests may compose
registries and catalogs directly with fakes.
## Adding A Module
## Adding An Extension
When adding a module, keep source-format and extraction-domain boundaries clear:
When adding a production module or validator:
- input modules may know external source formats;
- extract modules may know artifact semantics and prompt/schema assets;
- merge and normalize modules own raw output combination and reconciliation;
- output modules own serialization, not diagnostics or CLI reporting.
1. implement the stage or validator contract and package-local key;
2. expose and test its spec, constructor, and registration function;
3. keep format or domain parsing inside the concrete package;
4. add package-owned prompt/schema assets when the extension is LLM-backed;
5. register it in `internal/cli/catalog.go` and add a default chain only when
production policy requires one;
6. add resolution and composition coverage for capabilities, options,
references, and validation behavior;
7. update the selectable-key catalog in [Configuration](../config.md), the
relevant external contract, this inventory, and maintained examples when
user-visible behavior changes.
Update [Development](../development.md), [Configuration](../config.md),
internal docs, integration docs, and examples when the new module becomes
implemented production behavior.
Do not add the extension to `docs/development.md`; that file routes by task and
does not inventory implementations.
## Tests To Inspect
- Package-local `*_test.go` files under the module or validator being changed.
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
composition.
- `internal/framework/pipeline/default_modules_test.go`: framework binding
defaults.
- `internal/cli/run_test.go`: production catalog, config resolution, and
end-to-end CLI composition.
- `internal/modules/sharedassets/**/*_test.go`: shared prompt and reference
assembly.

160
docs/internal/overview.md
View File

@@ -1,127 +1,105 @@
# Internal Overview
This document maps the implemented Notarius components and their ownership. It
complements the durable invariants in [Architecture](../policy/architecture.md)
and links to focused internal documentation for deeper behavior.
This document inventories the implemented Notarius components. Normative
boundaries and dependency direction belong in
[Architecture](../policy/architecture.md); external behavior belongs in the
[CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [integration contracts](../integrations/).
## Execution Path
The executable delegates to the CLI, which resolves configuration and wires the
production application around the framework runner:
`cmd/notarius` delegates to `internal/cli`, the production composition root.
The CLI loads configuration, builds the production catalogs and runtime
collaborators, invokes `internal/framework/pipeline`, and places the logical
output files returned by the runner. Diagnostics, checkpoints, and debug
recorders are optional side-channel collaborators supplied at this boundary.
```text
cmd/notarius
-> internal/cli
-> config resolution + production registries + LLM client
-> input -> chunk -> extract -> merge -> normalize -> output
-> durable output writes
Pipeline side channels:
diagnostics checkpoints debug artifacts
```
Pipeline execution is serial. Configuration selects modules for the fixed stage
shape; registries construct them after profile, capability, validator, and
reference resolution.
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
a sorted set of artifact lanes before the runner constructs any stage module.
## Application Boundary
`cmd/notarius` contains the executable entry point and delegates process exit
behavior to `internal/cli`.
`internal/cli` owns command parsing, configuration discovery, production module
and validator registration, prompt asset collection, production LLM client
construction, reference preparation, workspace recorder setup, durable output
writes, and user-facing stdout, stderr, and exit codes. It is the composition
root for concrete production packages.
| Package | Implemented responsibility |
| --- | --- |
| `cmd/notarius` | Executable entry point and process exit delegation. |
| `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
## Core Packages
| Package | Implemented responsibility |
| --- | --- |
| `internal/core/artifacts` | Run manifests and artifact serialization shapes. |
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline configuration. |
| `internal/core/diagnostics` | Diagnostics run directories, artifact writers, atomic writes, and retention decisions. |
| `internal/core/source` | Generic source documents, units, references, and validation. |
| `internal/core/workspace` | Workspace settings, safe paths and writes, checkpoint identities, and checkpoint manifest types. |
These packages provide concrete, deterministic models and policy. Production
module registration occurs at the CLI boundary rather than in core packages.
| `internal/core/artifacts` | Run-manifest and provenance models. |
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
| `internal/core/source` | Generic source documents, units, references, lookup, and validation. |
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
## Framework Packages
| Package | Implemented responsibility |
| --- | --- |
| `internal/framework/contracts` | Stage, validator, reference, output, and structured LLM interfaces and request/result types. |
| `internal/framework/pipeline` | Module registries, profile resolution, capability checks, reference materialization, validation chains, retries, orchestration, warnings, and manifest population. |
| `internal/framework/contracts` | Stage, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema asset registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload envelopes. |
| `internal/framework/debug` | Workspace-backed framework and LLM debug artifacts. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
Framework contracts carry raw stage outputs between modules. The runner owns
provenance, validation sequencing, rejection handling, checkpoint boundaries,
debug boundaries, and final manifest assembly.
Framework contracts carry raw stage results between implementations. The
runner owns handoff provenance, validation sequencing, rejection handling,
checkpoint and debug boundaries, and final manifest assembly.
## Production Modules
## Production Extensions
Production implementations live under `internal/modules` and register through
the CLI catalog.
The canonical catalogs of user-selectable
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) keys are in
Configuration. The implemented module packages are:
| Stage | Module key | Package | Role |
| --- | --- | --- | --- |
| Input | `seriatim` | `internal/modules/input/seriatim` | Converts Seriatim transcript JSON into the generic source model. |
| Chunk | `generic` | `internal/modules/chunk/generic` | Splits ordered source units by configured unit counts and overlap. |
| Chunk | `dnd/scenes` | `internal/modules/chunk/dnd/scenes` | Uses structured LLM output to create contiguous D&D scene chunks. |
| Extract | `dnd/spells` | `internal/modules/extract/dnd/spells` | Extracts source-grounded D&D spell-cast artifacts. |
| Merge | `appendorder` | `internal/modules/merge/appendorder` | Combines accepted extract outputs in chunk order. |
| Normalize | `noop` | `internal/modules/normalize/noop` | Preserves accepted merged output unchanged. |
| Output | `json` | `internal/modules/output/json` | Encodes manifests, indexes, warnings, rejections, and accepted lane payloads as logical JSON files. |
| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/input/seriatim` | Parses the supported Seriatim transcript format into the generic source model. |
| `internal/modules/chunk/generic` | Splits ordered source units by unit count and overlap. |
| `internal/modules/chunk/dnd/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| `internal/modules/extract/dnd/spells` | Produces source-grounded D&D spell-cast raw output. |
| `internal/modules/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
`internal/modules/sharedassets` composes shared prompt filesystems.
`internal/modules/sharedassets/dnd` owns shared D&D prompt fragments, reference
slots, prompt input assembly, and source-unit reference helpers.
`internal/modules/sharedassets/dnd` owns reusable D&D prompt fragments,
reference declarations, prompt input assembly, and source-unit reference
helpers.
## Validators
Concrete validators live under `internal/validators`. Generic packages provide
unconditional test decisions, JSON syntax validation, and JSON Schema
validation. D&D spell packages provide shape, source-reference, and
source-relatedness decisions, with `spellpayload` holding their shared parser
and lookup helpers. Production chain composition is owned by `internal/cli`.
Concrete validators live under `internal/validators` and register separately
from stage modules. Generic validators cover unconditional test decisions, JSON
syntax, and JSON Schema. D&D spell validators cover artifact shape, source
reference validity, and source relatedness.
Implementation details for all production extensions are in
[Module Internals](modules.md).
The production default chain for `dnd/spells` extract output is registered
centrally in `internal/cli`; module packages produce raw output but do not own
the production approve/reject policy.
## Run-State Components
## Files And Run State
Notarius keeps distinct output and inspection surfaces:
| Surface | Owner | Purpose |
| Surface | Implemented owners | Internal purpose |
| --- | --- | --- |
| Durable output | Output module and CLI writer | User-consumable run files. |
| Diagnostics | `internal/core/diagnostics` and CLI | Redacted run inspection, reports, warnings, and failures. |
| Checkpoints | `internal/framework/checkpoint` | Validated stage reuse for explicit resume. |
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Sensitive framework-boundary and LLM call inspection. |
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
Workspace settings determine whether and where diagnostics, checkpoints, and
debug artifacts are written. Concrete stage modules do not receive workspace
paths.
Physical layout, retention, recovery, and sensitive-data handling are defined
in [Operations](../operations.md). Concrete stage modules receive recorder
interfaces and request data, not workspace paths.
## Focused Internal Documentation
## Focused Documentation
- [Pipeline Internals](pipeline.md): resolution, execution, validators,
references, retries, checkpoints, outputs, and manifests.
- [Module Internals](modules.md): production module contracts, capabilities,
options, prompts, schemas, and registration.
- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
checkpoint/debug hooks, and result assembly.
- [Module Internals](modules.md): production modules, validators, assets,
registration, and the contributor recipe for adding an extension.
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
assets, scheduling, profile recording, and redaction.
- [Diagnostics Internals](diagnostics.md): diagnostics files, retention,
failure behavior, and path safety.
## Test Surfaces
The repository uses focused package tests, registry and pipeline composition
tests, a fake-backed walking skeleton, fixture-driven CLI coverage, and local
test servers for LLM integration behavior. Tests do not require real provider
calls.
- [Diagnostics Internals](diagnostics.md): scoped writers, retention
coordination, CLI failure flow, and path safety.

396
docs/internal/pipeline.md
View File

@@ -1,295 +1,185 @@
# Pipeline Internals
The implemented pipeline runner lives in `internal/framework/pipeline`. It
executes the fixed workflow defined by the architecture policy:
The implemented resolver and runner live in `internal/framework/pipeline`.
Their fixed workflow and ownership boundaries are defined by
[Architecture](../policy/architecture.md#system-shape). Configuration fields,
defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).
```text
input -> chunk -> extract -> merge -> normalize -> output
```
Pipeline execution is serial. Resolution fixes the selected lanes and all
stage bindings before the runner constructs stage implementations.
Pipeline execution is serial. The runner executes the resolved lanes one after
another in the fixed workflow order.
## Resolution
## Profile Resolution
`internal/core/config.Config.Resolve` validates the loaded configuration,
selects the named profile, applies the runtime inputs supplied by the CLI, and
calls `pipeline.ResolvePipeline`.
Config loading produces `pipeline.PipelineProfile` values. Resolution happens
before execution:
`ResolvePipeline`:
1. `internal/core/config.Config.Resolve` validates config and finds the named
pipeline.
2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
3. Module bindings are defaulted:
- chunk: `generic`
- merge: `appendorder`
- normalize: `noop`
- output: `json`
- LLM profile: empty, which lets Scriptorium prompt defaults choose a
profile.
4. The module catalog is checked for each bound module key.
5. Module capabilities are checked in workflow order.
6. A digest is calculated from the resolved pipeline without the digest field.
1. selects and sorts artifact lanes;
2. completes omitted bindings using the documented configuration defaults;
3. looks up each module and validator spec without constructing it;
4. checks required and provided capabilities in workflow order;
5. resolves target-aware reference bindings and validator chains;
6. calculates a digest over the resolved structure.
The CLI writes the resolved pipeline and digest to diagnostics.
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
bindings, validator chains, reference targets, and the digest. It does not read
reference bytes or construct runtime modules. CLI lane and reference selector
syntax is defined in the [CLI reference](../cli.md#run).
Pipeline profiles and artifact lanes may include reference binding maps keyed by
reference slot name. During resolution, pipeline-level bindings act as defaults
for selected chunk, extractor, merger, and normalizer targets that declare the
slot; target-local bindings override or add bindings for that target. Runtime
`--reference` requests override target config bindings, and runtime unbinds
remove optional target bindings. Flat runtime slot names are resolved only when
exactly one selected target declares the slot; otherwise the CLI requires a more
specific selector such as `chunk.slot`, `lane.extract.slot`,
`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings
against the declaring target specs and stores the bindings in target-aware
resolved reference holders. It does not read reference files or include
reference bytes in source digests.
## Reference Materialization
During run preparation, resolved file references for chunk, extractor, merger,
and normalizer targets are materialized before any LLM-backed pipeline work. Config
bindings resolve relative to the config file, and CLI bindings resolve relative
to the current working directory. Materialization accepts UTF-8 text files,
computes `sha256:` content digests, records file origins, infers canonical base
media types from file extensions, enforces declared byte limits, and warns for
empty bound files. Media-type acceptance is checked only when a slot declares
`AcceptedMediaTypes`; unknown extensions are recorded as
`application/octet-stream`. Reference content is omitted from diagnostics and
manifests. The CLI writes provenance-only resolved reference diagnostics, and
the run manifest records target-stage reference provenance separately from
source digests. Runtime reference content is passed to the matching chunker,
extractor, merger, or normalizer request. LLM-backed modules pass that material
onward as named Scriptorium prompt inputs.
The CLI calls `MaterializeReferences` after resolution and before constructing
the LLM client or running the pipeline. The materializer checks each binding
against its resolved target declaration, reads and validates the file, and
builds both a `contracts.ReferenceSet` and provenance-only metadata on the
corresponding `ResolvedReferenceTarget`.
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
those bytes into the source document. Chunk, merge, and normalize requests
receive the original source material as `SourceInput`; extraction requests
receive chunk-scoped source material built from the current `SourceChunk`
content, media type, and origin metadata. The raw input payload is not written
to manifests or default diagnostics.
The runner clones the resulting set into the chunk, extract, merge, or normalize
request that owns the target. LLM-backed extensions may convert those items into
named prompt inputs. Reference content remains separate from source evidence and
source digests.
The CLI also carries an optional run `session_id`. The runner makes it available
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
through their structured completion requests so Scriptorium can include it in
prompt execution metadata.
Binding precedence, path resolution, accepted content, and media-type behavior
are configuration contracts; see [Configuration](../config.md#pipelines).
Durable provenance is defined in the
[JSON output contract](../integrations/json-output.md#manifestjson), while
runtime sensitive-data handling belongs in [Operations](../operations.md).
When workspace resume checkpointing is enabled, the CLI constructs a checkpoint
recorder after pipeline resolution and reference materialization and passes it
through `pipeline.RunInput`. The runner records source, chunk, extract, merge,
and normalize outcomes through that interface. Concrete modules do not receive
workspace paths and do not write checkpoint files directly.
## Registries And Specs
For `run --resume`, the CLI also passes a checkpoint loader. The runner consults
the loader in workflow order and reuses only checkpoints whose manifest schema,
status, identity digest, dependency fingerprints, payload files, and payload
digests validate for the current invocation. The identity includes the resolved
pipeline, selected lanes, source/input digest, runtime overrides that affect
execution, and materialized reference digests. Missing or invalid checkpoints
fall back to normal execution and are refreshed by the recorder.
`pipeline.Registries` holds constructors used during execution.
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
resolution. Separate registries exist for every stage and for validators;
`ValidatorChainRegistry` stores production default-chain mappings.
When workspace debug output is enabled, the CLI passes a debug recorder for the
current run ID. The runner writes framework-boundary inputs, outputs,
structured LLM calls, validator calls, timing, and retry attempt metadata
through that interface. Each retry or validator attempt records any LLM calls
made within that attempt in an `llm_calls` array and writes paired
`prompt-000N.json` and `response-000N.json` metadata files under the attempt
directory. LLM response bodies are written as sibling `response-content-000N.*`
files, using pretty-printed JSON when the content is valid JSON and raw text
otherwise. Debug output is not used for resume and can contain sensitive source,
reference, prompt, and model-output material. Concrete modules still do not
receive workspace paths.
A `ModuleSpec` declares its stage plus required and provided capabilities.
Chunk, extract, merge, and normalize specs may also declare reference slots.
Registry implementations defensively copy spec metadata, reject duplicate keys,
and verify that a constructed implementation reports the registered key.
## Registries And Module Specs
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
the execution class to reject incompatible profile bindings before execution.
The current production catalog and default chain are listed only in
[Configuration](../config.md#implemented-production-validators).
`pipeline.Registries` holds concrete constructors for execution. A
`pipeline.ModuleCatalog` exposes module specs for config validation and
resolution. The catalog also exposes validator specs and central default
validator-chain mappings without constructing modules or validators.
## Runner Boundary
Every production module registers a `ModuleSpec` with:
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
LLM client, run identity and timing, optional session and profile metadata, and
checkpoint/debug collaborators. The runner parses source bytes through the
selected input adapter. Later stage requests receive the generic source model;
extract requests receive chunk-scoped input material, while chunk, merge, and
normalize requests retain access to the original source material.
- `Key`: module key used in config;
- `Stage`: module kind such as input, chunk, extract, merge, normalize,
validate, or output;
- `Provides`: capabilities added after that module runs;
- `Requires`: capabilities that must already be available.
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
rejected results, warnings, checkpoint events, and logical files returned by the
output encoder. The CLI owns diagnostics and durable filesystem writes after the
runner returns.
Chunk, extract, merge, and normalize specs may also declare reference slots. Slot
declarations are available from registry metadata without constructing module
instances. Input, validate, and output specs must not declare reference slots.
Capability checks prevent incompatible pipeline composition before a run starts.
Every production validator registers a `ValidatorSpec` with:
- `Key`: validator key used in config and manifests;
- `ExecutionClass`: `deterministic` or `llm_backed`.
Default validator chains are keyed by workflow stage and module key. Production
currently registers a default chain for `extract` module `dnd/spells` only.
## Runner Input And Output
`pipeline.RunInput` carries:
- a `ResolvedPipeline`;
- optional source ID, input path, and raw input bytes;
- a structured LLM client;
- run ID, start time, LLM profile manifest metadata, and CLI metadata.
`pipeline.RunOutput` carries:
- run manifest;
- normalized raw outputs;
- rejected raw outputs;
- warnings;
- logical output files returned by the output encoder.
The CLI owns durable file writes and diagnostics writes after the runner returns.
## Execution
## Execution Flow
The runner:
1. validates run input and registries;
2. builds the input adapter and parses the raw input into a source document;
3. validates the source document;
4. builds the chunker and produces source chunks, retrying when configured;
5. validates source chunks against framework invariants and the resolved chunk
validator chain;
6. runs each selected artifact lane in sorted resolved order;
7. builds the output encoder and validates logical output file names.
8. passes accepted normalized raw outputs, rejected output records, warnings,
and the manifest to the output encoder.
1. validates its input and registries;
2. builds the input adapter, parses the raw input, and validates the generic
source document;
3. obtains or executes the chunk result;
4. validates and canonicalizes chunks;
5. executes each resolved artifact lane in order;
6. builds the output encoder and validates its logical file results;
7. returns the assembled manifest, outcomes, warnings, and files.
## Chunk Results
Within each artifact lane, it builds the extractor, merger, and normalizer,
then performs these transitions:
Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
with the validated source document, reference set, structured LLM client, the
configured LLM profile, module options, and run metadata. Deterministic and
LLM-backed chunkers use the same contract; provider construction stays outside
chunk modules.
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
provenance;
2. validate each raw extract result and omit rejected results from merge input;
3. skip the rest of the lane when no extract result is accepted;
4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
6. normalize the accepted merge result;
7. validate and append the accepted normalized result.
When chunking succeeds, the runner validates generic chunk invariants before
running extractors:
Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.
- chunk IDs must be non-empty and unique in the chunk result;
- each chunk `SourceID` must match the source document ID;
- each chunk `Index` must match its zero-based returned order;
- each chunk start and end unit ID must exist in the source document, with the
start unit at or before the end unit;
- each chunk must include non-empty extraction content and media type;
- each chunk must contain at least one source unit;
- a chunk must not repeat a source unit;
- every chunk source unit must exist in the source document;
- source units inside each chunk must appear in source-document order.
## Chunk Canonicalization
After validation, the runner rebuilds each chunk from source-document units by
integer ID, preserving chunk boundaries, content bytes, media type, and cloned
chunk metadata. Extractors and downstream stages therefore see canonical source
units, while `SourceChunk.Metadata` remains the supported place for
chunker-owned context.
Before lane execution, generic validation requires unique chunk IDs, matching
source identity, indexes matching returned order, valid ordered boundaries,
non-empty content and media type, and at least one valid source unit per chunk.
Units may not repeat inside a chunk and must preserve source-document order.
If chunk validation rejects the chunk result after configured retries, the runner
records a rejected raw output and skips downstream lane execution. Framework-level
chunking or validation errors that remain after configured retries fail the run.
The runner then rebuilds each chunk's unit slice from the source document by
unit ID. It preserves the module-owned boundaries, content, media type, and
cloned metadata. The framework permits gaps and overlap between separate
chunks; stricter coverage policy belongs to the chunk implementation.
The framework does not require complete source-unit coverage and does not reject
overlap between different chunks. Stricter policies, such as full coverage or
non-overlap, belong to individual chunk modules when they are part of that
module's contract.
## Validation And Retries
Within an artifact lane, the runner:
Chunk, extract, merge, and normalize results pass through the resolved validator
chain for their stage and module. Each validator receives the raw payload plus
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
and run context. Validators execute in resolved order and stop at the first
error or rejection. An empty chain approves the result.
1. builds the extractor, merger, and normalizer;
2. records module manifest metadata when modules provide it;
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when
configured;
4. fills runner-owned provenance on each extract output, including lane ID,
extractor key, source ID, chunk ID, and chunk index;
5. validates raw extract outputs and omits rejected outputs from merge input;
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
when configured;
7. validates raw merge output and skips normalization for rejected merge output;
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
retrying when configured;
9. validates raw normalize output and appends accepted normalized raw output to
`RunOutput.NormalizeOutputs`.
`runWithRetry` performs the initial module call plus the configured additional
attempts. Each attempt includes module execution and its complete validation
chain. A module or validator error retries and becomes a framework error after
the final attempt. A rejection retries and becomes a recorded `RejectedOutput`
after the final attempt. Cancellation stops retry processing immediately.
## Validators
Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
from discarded attempts are not promoted. Configuration owns retry counts and
validator overrides; see [Module Bindings](../config.md#module-bindings).
The runner handoff is raw-output based. Chunkers, extractors, mergers, and
normalizers do not advertise validator chains through their module interfaces.
Resolved validation chains receive the raw module output plus stage, lane,
module, source, chunk, schema, session, reference, LLM client/profile, binding
option, and run metadata context. Chunk validators receive the chunk result
collection, merge validators receive the ordered extract outputs used by the
merge, and normalize validators receive the accepted merge output. Empty chains
approve output by default. Response-schema provenance may include in-memory JSON
schema bytes for validators. Those bytes are omitted from manifests,
diagnostics, and encoded output files.
## Checkpoint And Debug Hooks
Resolved validator chains come from central default mappings unless a
stage-local config override is set on `chunk`, lane `extract`, lane `merge`, or
lane `normalize`. Explicit empty overrides are valid and are recorded as empty
chains in manifests. Explicit non-empty overrides replace the default chain and
preserve configured order.
The runner depends on recorder and loader interfaces, using no-op
implementations when collaborators are absent. Each checkpointed workflow
boundary records a running, succeeded, or failed transition. Reuse decisions
are consulted in workflow order and accepted payloads are cloned before
entering the normal handoff path. Dependency fingerprints connect later
checkpoints to the exact accepted results on which they depend.
The production default chain for `extract` module `dnd/spells` is:
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
boundaries. Context scopes associate nested LLM calls with the module or
validator attempt that made them. Debug-write failures are framework errors;
debug data is never used as a checkpoint source.
1. `generic/valid_json`
2. `generic/valid_json_schema`
3. `extract/dnd/spells/shape`
4. `extract/dnd/spells/source_refs`
5. `extract/dnd/spells/source_relatedness`
Checkpoint identity, physical layout, reuse behavior, and debug artifact
handling are operator contracts in [Operations](../operations.md). Serialization
and recorder implementation are inventoried in
[Internal Overview](overview.md#run-state-components).
No other production module currently has a default validator chain.
## Results And Failures
Validator rejection is a non-fatal run outcome: the rejected output is recorded
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
errors are framework-level errors and retry according to the relevant binding.
Warning-only validators return approved results with warnings; those warnings
are promoted only from successful attempts whose outputs are used.
The runner owns manifest assembly and handoff summaries but not the durable JSON
schema. It records resolved module and lane provenance, validator chains,
source/reference identities, selected LLM profiles, normalized and rejected
summaries, status, and timing. Raw payload bytes remain outside the manifest.
Module metadata providers may add non-secret singleton or lane-scoped metadata.
## Warnings And Failures
Execution errors include stage, module, lane, or validator context. Once a
manifest exists, a failing run returns it with failed status and completion
time. Successful status reflects whether any raw result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).
Warnings from the successful chunking, extraction, merging, and normalization
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
with output encoder warnings. Warnings from discarded retry attempts are not
promoted to final warnings.
## Tests To Inspect
Errors wrap the operation and module key or lane context. If execution fails
after a manifest exists, the returned manifest is marked `failed` and receives a
completion timestamp.
On successful execution, the manifest validation status is:
- `approved` when no raw outputs were rejected;
- `rejected` when at least one raw output was rejected.
## Manifest Population
The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level
module metadata, artifact lanes, LLM profile metadata, source digest,
reference provenance, normalized raw output summaries, rejected output
summaries, validation status, and timing. Raw output summaries include lane ID,
normalizer module key, media type, source ID, and response-schema provenance
when present. Rejected output summaries include stage, lane, module, chunk,
validator or reason, message, attempt count, and optional diagnostic artifact
path. The manifest does not include raw output payload bytes.
Singleton pipeline modules may add non-secret metadata by implementing
`contracts.ManifestMetadataProvider`. The runner records that metadata under
`module_metadata` with stable keys for `input`, `chunker`, and `output`.
Lane-owned modules may add non-secret metadata through
`artifact_lanes[].metadata`. The runner records extractor, merger, and
normalizer metadata there. The D&D spell extractor uses lane metadata for
prompt and response-schema provenance.
## JSON Output
The production JSON output encoder writes `manifest.json`, `index.json`,
`warnings.json`, `rejected.json`, and one pretty-printed JSON file per accepted
normalized lane output under `lanes/`. It accepts only normalized outputs with
valid `application/json` payloads. Unsupported media types, invalid JSON, unsafe
logical paths, and duplicate sanitized lane file names fail the run before
durable output files are written.
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
capabilities, validator chains, and digest behavior.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
rejections, warnings, checkpoints, debug hooks, and manifests.
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
workflow composition.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
collaborators.

22
docs/policy/architecture.md
View File

@@ -32,21 +32,17 @@ Notarius is contract-first without being abstraction-heavy. Interfaces and
extension points should protect demonstrated boundaries. New abstraction is not
itself an architectural goal.
## Package Layout And Dependency Direction
## Layers And Dependency Direction
| Area | Ownership |
| --- | --- |
| `cmd/notarius` | Executable entry point; delegates to the CLI. |
| `internal/cli` | Application boundary, production composition, runtime setup, durable writes, and user-facing results. |
| `internal/core` | Generic deterministic models and policy for source material, configuration, manifests, diagnostics, and workspace identity. |
| `internal/framework` | Reusable contracts, registries, pipeline orchestration, validation mechanics, checkpoints, debug boundaries, and LLM runtime plumbing. |
| `internal/modules` | Concrete pipeline stage behavior. |
| `internal/validators` | Concrete approve/reject policies. |
The application boundary is the composition root and may depend on concrete
implementations. Domain-neutral model and framework layers provide reusable
policy, contracts, and orchestration. Concrete input, pipeline, output, and
validation extensions depend inward on those generic layers.
The CLI is the composition root and may import concrete implementations. Core
and framework packages cooperate as generic application layers; neither may
depend on production modules or validators. Concrete implementations may depend
on core models and framework contracts.
Generic layers must not depend on production extensions. Concrete extensions
must not compose the application or take ownership of process behavior. The
current packages implementing these layers are inventoried in
[Internal Overview](../internal/overview.md).
The following dependency boundaries are mandatory: