320 lines
13 KiB
Markdown
320 lines
13 KiB
Markdown
# Architecture
|
|
|
|
This document defines the development principles for Notarius. It is
|
|
inward-facing: developers and LLM coding agents should use it to preserve the
|
|
project's shape, boundaries, and invariants as the code evolves.
|
|
|
|
## Project Shape
|
|
|
|
Notarius is a small, explicit, dependency-light Go application for extracting
|
|
structured artifacts from source material using modular extractors.
|
|
|
|
The application should be contract-first but not abstraction-heavy. Add
|
|
interfaces and extension points when they protect a real boundary:
|
|
|
|
- external source formats;
|
|
- extractor modules;
|
|
- validators;
|
|
- LLM providers and runtime plumbing;
|
|
- output schemas and embedded assets.
|
|
|
|
Avoid abstractions that only anticipate hypothetical complexity. Prefer narrow
|
|
contracts that can be exercised by tests and real modules.
|
|
|
|
## Core Invariants
|
|
|
|
The core framework must remain source-agnostic and domain-agnostic.
|
|
|
|
Source-format details belong in input modules. Transcript-specific concepts
|
|
such as segments, speakers, timestamps, and transcript schemas must not spread
|
|
into runner, extractor, or validator framework code.
|
|
|
|
Extraction-domain details belong in extract modules. D&D-specific concepts
|
|
such as spells, NPCs, items, combat turns, and encounters must not spread into
|
|
core source, runner, or LLM framework packages.
|
|
|
|
Extracted facts should be grounded with source references. Source references
|
|
should point to generic source units, not to transcript-only structures.
|
|
|
|
The application workflow is:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
These stages should remain explicit in the architecture. Chunking, merging, and
|
|
normalization must not be hidden inside domain extract modules when they represent
|
|
general pipeline behavior.
|
|
|
|
## Dependency Policy
|
|
|
|
Prefer the Go standard library where practical.
|
|
|
|
Use external dependencies only when justified by correctness, security,
|
|
interoperability, or substantial complexity reduction. Good reasons include
|
|
widely used file formats, complex validation behavior, or secure transport
|
|
handling.
|
|
|
|
Avoid dependencies for small conveniences. Do not let external dependency types
|
|
leak across internal package boundaries unless the dependency is itself the
|
|
explicit public contract of that package.
|
|
|
|
## Package Layout
|
|
|
|
Use this layout unless a change documents a better project-specific reason.
|
|
|
|
CLI and executable entrypoint:
|
|
|
|
- `cmd/notarius`: executable entrypoint.
|
|
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
|
|
|
|
Core deterministic model and policy:
|
|
|
|
- `internal/core/config`: configuration structs, defaults, loading, precedence, and validation.
|
|
- `internal/core/source`: source document, source unit, and source reference types.
|
|
- `internal/core/sourcechunking`: deterministic chunking of ordered source units.
|
|
- `internal/core/artifacts`: artifact envelope, artifact candidates, rejected artifacts, and manifests.
|
|
- `internal/core/diagnostics`: run directories and diagnostics artifact paths.
|
|
- `internal/core/reporting`: process reports and report serialization.
|
|
- `internal/core/inputcatalog`: known input adapter keys and metadata.
|
|
- `internal/core/extractorcatalog`: known extractor keys and metadata.
|
|
|
|
Reusable framework plumbing:
|
|
|
|
- `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
|
|
- `internal/framework/runner`: orchestration across adapters, extractors, validators, and artifact output.
|
|
- `internal/framework/pipeline`: shared pipeline-stage orchestration types, when needed.
|
|
- `internal/framework/extraction`: shared extraction helper code.
|
|
- `internal/framework/merge`: shared merge-stage behavior.
|
|
- `internal/framework/normalize`: shared normalization-stage behavior.
|
|
- `internal/framework/output`: output encoding contracts and shared helpers.
|
|
- `internal/framework/validators`: shared validator runtime behavior and decision checks.
|
|
- `internal/framework/llm`: LLM runtime, scheduling, and provider adapters.
|
|
- `internal/framework/responseschema`: embedded structured-output schema registry.
|
|
- `internal/framework/structuredoutput`: structured-output parsing and malformed-response handling.
|
|
- `internal/framework/promptcontext`: source-document prompt rendering helpers.
|
|
- `internal/framework/warnings`: shared warning records.
|
|
|
|
Domain implementations:
|
|
|
|
- `internal/modules/input/<name>`: input-stage modules that parse external input into core source documents.
|
|
- `internal/modules/chunk/<name>`: chunk-stage modules.
|
|
- `internal/modules/extract/<domain>/<name>`: extract-stage extractor modules.
|
|
- `internal/modules/merge/<name>` or `internal/modules/merge/<domain>/<name>`: merge-stage modules.
|
|
- `internal/modules/normalize/<name>` or `internal/modules/normalize/<domain>/<name>`: normalize-stage modules.
|
|
- `internal/modules/output/<name>`: output-stage modules.
|
|
- `internal/validators/<validator>`: built-in validator implementations.
|
|
- `internal/prompts`: embedded prompt assets and prompt metadata registry.
|
|
- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
|
|
|
|
Package-private implementation constants may live near the package that owns
|
|
them, preferably in `constants.go` when useful.
|
|
|
|
## Stage Modules
|
|
|
|
Concrete business logic should live under `internal/modules/<stage>/...`.
|
|
|
|
Stage-oriented module layout is preferred because it makes the application
|
|
workflow visible in the filesystem:
|
|
|
|
```text
|
|
internal/modules/input/...
|
|
internal/modules/chunk/...
|
|
internal/modules/extract/...
|
|
internal/modules/merge/...
|
|
internal/modules/normalize/...
|
|
internal/modules/output/...
|
|
```
|
|
|
|
Use short, lowercase, idiomatic Go package names. Prefer names such as
|
|
`dndtranscript`, `appendorder`, and `spells` over names like `dnd_transcript`,
|
|
`serial_merge`, or `spell_extractor` that repeat parent-stage context.
|
|
|
|
## Input Modules
|
|
|
|
Use a hexagonal architecture style for source input.
|
|
|
|
Input modules translate external source formats into the core source model.
|
|
They may know about external schema details, source-specific metadata, and
|
|
format-specific validation rules. They should not own extraction-domain
|
|
decisions.
|
|
|
|
Other packages should interact with source input through adapter contracts and
|
|
core source types. Input module implementation details and external dependency
|
|
types must not leak into framework or extract module packages.
|
|
|
|
Input module metadata may preserve source-specific facts such as transcript speaker,
|
|
timestamps, Markdown heading path, page number, or block ID. Framework code may
|
|
carry metadata through, but should not require a specific adapter's metadata
|
|
shape.
|
|
|
|
## Extractors
|
|
|
|
Extractors are independent modules that process source chunks or whole source
|
|
documents and produce one kind of structured artifact candidate.
|
|
Each extract module owns:
|
|
|
|
- its artifact semantics;
|
|
- its prompt usage;
|
|
- its structured response schema selection;
|
|
- its validator chain;
|
|
- any domain-specific mapping or interpretation.
|
|
|
|
Extract modules should depend on framework contracts and core source/artifact
|
|
types. They should not depend on concrete input module packages.
|
|
|
|
Extractors should not be the only place where chunking, merging, or
|
|
normalization happens. They may choose processing mode or provide domain-specific
|
|
merge/normalization behavior when generic behavior is insufficient, but the
|
|
pipeline stages themselves are framework concepts.
|
|
|
|
The runner should be able to compose, skip, resume, or run individual extractors
|
|
when their prerequisites are satisfied. Ordering should be explicit through
|
|
configuration, a default sequence, or documented orchestration rules.
|
|
|
|
Extractor selection must go through a registry or equivalent mechanism rather
|
|
than scattered conditionals.
|
|
|
|
## Pipeline Stages
|
|
|
|
The pipeline has six conceptual stages:
|
|
|
|
1. input: external source material becomes a `SourceDocument`;
|
|
2. chunk: a `SourceDocument` becomes ordered source chunks;
|
|
3. extract: extractors produce artifact candidates from chunks or whole documents;
|
|
4. merge: per-chunk candidates become a merged candidate collection;
|
|
5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues;
|
|
6. output: final artifacts are serialized.
|
|
|
|
Chunking is first-class because source documents may be too large for a single
|
|
LLM pass. Chunkers should preserve source-unit order and produce stable chunk
|
|
metadata.
|
|
|
|
Merge and normalize are separate concerns. Merge combines per-chunk results into
|
|
a deterministic collection. Normalize performs semantic reconciliation after
|
|
merge. Generic append-in-chunk-order merge and no-op normalization should be
|
|
available for simple artifact types, while domain-specific behavior can be
|
|
provided where needed.
|
|
|
|
The framework should allow serial and parallel chunk processing. The first
|
|
implementation may execute chunks serially for determinism, but contracts should
|
|
not prevent later parallel execution.
|
|
|
|
## Validators
|
|
|
|
Validators should be independently testable and composable.
|
|
|
|
Deterministic validators should run before LLM-backed validators when both are
|
|
present. Validator decision semantics should be explicit: each candidate
|
|
artifact should receive exactly one decision from each validator that evaluates
|
|
it.
|
|
|
|
Shared validator runtime mechanics belong under `internal/framework/validators`.
|
|
Concrete validator behavior belongs under `internal/validators/<validator>`.
|
|
|
|
## LLM Runtime
|
|
|
|
LLM provider details belong behind transport-neutral framework contracts.
|
|
|
|
Provider-specific HTTP request and response types should stay inside the LLM
|
|
runtime package. Prompt construction should stay in extractors, validators, or
|
|
shared prompt-context helpers; provider adapters should not own domain prompt
|
|
logic.
|
|
|
|
Errors, diagnostics, reports, and redacted config must not expose secrets.
|
|
|
|
## Configuration
|
|
|
|
Centralize configuration loading, processing, precedence, defaults, and
|
|
validation in `internal/core/config`.
|
|
|
|
The goal is to make configuration discoverable and avoid implicit or hidden
|
|
operational values. User-visible defaults and cross-package operational defaults
|
|
should be defined in config code.
|
|
|
|
Unless documented otherwise, precedence is:
|
|
|
|
1. CLI flags
|
|
2. environment variables
|
|
3. configuration file
|
|
4. built-in defaults
|
|
|
|
Prefer YAML configuration unless the project has a strong reason to use another
|
|
format. Config files should be discoverable at
|
|
`/usr/local/etc/notarius/config.yml`, with a CLI override via `--config`.
|
|
|
|
Configuration files should not contain raw secrets unless the application is
|
|
explicitly designed for that. Prefer environment variables or secret files for
|
|
secrets.
|
|
|
|
Stage-module-specific configuration should remain grouped by the module that
|
|
owns it.
|
|
|
|
## Embedded Assets
|
|
|
|
Store embedded JSON schemas, Markdown prompts, templates, and similar assets as
|
|
separate files, not inline string literals, unless there is a strong reason
|
|
otherwise.
|
|
|
|
Embedded prompts and response schemas should have stable IDs, versions, source
|
|
metadata, and hashes suitable for diagnostics and run manifests.
|
|
|
|
## Errors and Logging
|
|
|
|
Errors should be actionable and preserve context. Wrap errors with operation and
|
|
path/resource context. CLI code should convert internal errors into concise
|
|
user-facing messages.
|
|
|
|
Errors and logs must not expose secrets.
|
|
|
|
Use structured logging where practical. Logs should describe operations, paths,
|
|
external calls, retries, and failure causes, but should not include large source
|
|
or artifact payloads by default.
|
|
|
|
## Context, Timeouts, and Cancellation
|
|
|
|
Long-running operations should accept `context.Context`. External calls,
|
|
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
|
|
workflows should respect cancellation and timeouts.
|
|
|
|
## State, Files, and Safety
|
|
|
|
If the application writes durable state, writes should be atomic where
|
|
practical. Multi-step workflows should preserve enough state to support
|
|
inspection, retry, or resume after failure.
|
|
|
|
Code that deletes, moves, or overwrites files must use narrow, explicit paths.
|
|
Avoid broad parent-directory operations. Cleanup that can cause data loss must
|
|
be opt-in.
|
|
|
|
## Testing
|
|
|
|
Core logic should be testable without real external services. Use fakes,
|
|
fixtures, or local test doubles for input modules, extract modules, validators,
|
|
and LLM clients where practical.
|
|
|
|
Contract-first work should include fake implementations that prove interfaces
|
|
compose before real input modules or extract modules depend on them.
|
|
|
|
Config examples should be load-tested once config files exist. Important CLI
|
|
workflows should have parser or command tests. Adapter, extractor, and validator
|
|
contracts should have focused tests that do not require running the full
|
|
application unless end-to-end coverage is intentional.
|
|
|
|
## Documentation
|
|
|
|
Documentation should follow the project documentation policy. Keep user docs
|
|
focused on implemented behavior. Put future, planned, or aspirational work only
|
|
under `docs/roadmap/`.
|
|
|
|
Core documentation should use generic terms such as source document, source
|
|
unit, source reference, input adapter, extractor, chunker, merger, normalizer,
|
|
artifact, validator, and run manifest.
|
|
|
|
Source-format details belong in input module or integration docs.
|
|
Domain-specific extraction details belong in extract module or artifact docs.
|
|
|
|
When changing architecture, config, CLI behavior, stage modules, extractor
|
|
contracts, validator contracts, LLM runtime behavior, or artifact schemas, update
|
|
the relevant docs and examples in the same change.
|