254 lines
10 KiB
Markdown
254 lines
10 KiB
Markdown
# Architecture
|
|
|
|
This document defines the development principles for Notarius. It is
|
|
inward-facing: developers and LLM coding agents should use it to preserve the
|
|
project's shape, boundaries, and invariants as the code evolves.
|
|
|
|
## Project Shape
|
|
|
|
Notarius is a small, explicit, dependency-light Go application for extracting
|
|
structured artifacts from source material using modular extractors.
|
|
|
|
The application should be contract-first but not abstraction-heavy. Add
|
|
interfaces and extension points when they protect a real boundary:
|
|
|
|
- external source formats;
|
|
- extractor modules;
|
|
- validators;
|
|
- LLM providers and runtime plumbing;
|
|
- output schemas and embedded assets.
|
|
|
|
Avoid abstractions that only anticipate hypothetical complexity. Prefer narrow
|
|
contracts that can be exercised by tests and real modules.
|
|
|
|
## Core Invariants
|
|
|
|
The core framework must remain source-agnostic and domain-agnostic.
|
|
|
|
Source-format details belong in input adapters. Transcript-specific concepts
|
|
such as segments, speakers, timestamps, and transcript schemas must not spread
|
|
into runner, extractor, or validator framework code.
|
|
|
|
Extraction-domain details belong in extractor packages. D&D-specific concepts
|
|
such as spells, NPCs, items, combat turns, and encounters must not spread into
|
|
core source, runner, or LLM framework packages.
|
|
|
|
Extracted facts should be grounded with source references. Source references
|
|
should point to generic source units, not to transcript-only structures.
|
|
|
|
## Dependency Policy
|
|
|
|
Prefer the Go standard library where practical.
|
|
|
|
Use external dependencies only when justified by correctness, security,
|
|
interoperability, or substantial complexity reduction. Good reasons include
|
|
widely used file formats, complex validation behavior, or secure transport
|
|
handling.
|
|
|
|
Avoid dependencies for small conveniences. Do not let external dependency types
|
|
leak across internal package boundaries unless the dependency is itself the
|
|
explicit public contract of that package.
|
|
|
|
## Package Layout
|
|
|
|
Use this layout unless a change documents a better project-specific reason.
|
|
|
|
CLI and executable entrypoint:
|
|
|
|
- `cmd/notarius`: executable entrypoint.
|
|
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
|
|
|
|
Core deterministic model and policy:
|
|
|
|
- `internal/core/config`: configuration structs, defaults, loading, precedence, and validation.
|
|
- `internal/core/source`: source document, source unit, and source reference types.
|
|
- `internal/core/sourcechunking`: deterministic chunking of ordered source units.
|
|
- `internal/core/artifacts`: artifact envelope, artifact candidates, rejected artifacts, and manifests.
|
|
- `internal/core/diagnostics`: run directories and diagnostics artifact paths.
|
|
- `internal/core/reporting`: process reports and report serialization.
|
|
- `internal/core/inputcatalog`: known input adapter keys and metadata.
|
|
- `internal/core/extractorcatalog`: known extractor keys and metadata.
|
|
|
|
External source and provider adapters:
|
|
|
|
- `internal/adapters/input/<name>`: source-format adapters that parse external input into core source documents.
|
|
- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
|
|
|
|
Reusable framework plumbing:
|
|
|
|
- `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
|
|
- `internal/framework/runner`: orchestration across adapters, extractors, validators, and artifact output.
|
|
- `internal/framework/extraction`: shared extraction helper code.
|
|
- `internal/framework/validators`: shared validator runtime behavior and decision checks.
|
|
- `internal/framework/llm`: LLM runtime, scheduling, and provider adapters.
|
|
- `internal/framework/responseschema`: embedded structured-output schema registry.
|
|
- `internal/framework/structuredoutput`: structured-output parsing and malformed-response handling.
|
|
- `internal/framework/promptcontext`: source-document prompt rendering helpers.
|
|
- `internal/framework/warnings`: shared warning records.
|
|
|
|
Domain implementations:
|
|
|
|
- `internal/extractors/<domain>/<extractor>`: domain-specific extractor packages.
|
|
- `internal/validators/<validator>`: built-in validator implementations.
|
|
- `internal/prompts`: embedded prompt assets and prompt metadata registry.
|
|
|
|
Package-private implementation constants may live near the package that owns
|
|
them, preferably in `constants.go` when useful.
|
|
|
|
## Input Adapters
|
|
|
|
Use a hexagonal architecture style for source input.
|
|
|
|
Input adapters translate external source formats into the core source model.
|
|
Adapters may know about external schema details, source-specific metadata, and
|
|
format-specific validation rules. They should not own extraction-domain
|
|
decisions.
|
|
|
|
Other packages should interact with source input through adapter contracts and
|
|
core source types. Adapter implementation details and external dependency types
|
|
must not leak into framework or extractor packages.
|
|
|
|
Adapter metadata may preserve source-specific facts such as transcript speaker,
|
|
timestamps, Markdown heading path, page number, or block ID. Framework code may
|
|
carry metadata through, but should not require a specific adapter's metadata
|
|
shape.
|
|
|
|
## Extractors
|
|
|
|
Extractors are independent modules that produce one kind of structured artifact.
|
|
Each extractor package owns:
|
|
|
|
- its artifact semantics;
|
|
- its prompt usage;
|
|
- its structured response schema selection;
|
|
- its validator chain;
|
|
- any domain-specific mapping or interpretation.
|
|
|
|
Extractors should depend on framework contracts and core source/artifact types.
|
|
They should not depend on concrete input adapter packages.
|
|
|
|
The runner should be able to compose, skip, resume, or run individual extractors
|
|
when their prerequisites are satisfied. Ordering should be explicit through
|
|
configuration, a default sequence, or documented orchestration rules.
|
|
|
|
Extractor selection must go through a registry or equivalent mechanism rather
|
|
than scattered conditionals.
|
|
|
|
## Validators
|
|
|
|
Validators should be independently testable and composable.
|
|
|
|
Deterministic validators should run before LLM-backed validators when both are
|
|
present. Validator decision semantics should be explicit: each candidate
|
|
artifact should receive exactly one decision from each validator that evaluates
|
|
it.
|
|
|
|
Shared validator runtime mechanics belong under `internal/framework/validators`.
|
|
Concrete validator behavior belongs under `internal/validators/<validator>`.
|
|
|
|
## LLM Runtime
|
|
|
|
LLM provider details belong behind transport-neutral framework contracts.
|
|
|
|
Provider-specific HTTP request and response types should stay inside the LLM
|
|
runtime package. Prompt construction should stay in extractors, validators, or
|
|
shared prompt-context helpers; provider adapters should not own domain prompt
|
|
logic.
|
|
|
|
Errors, diagnostics, reports, and redacted config must not expose secrets.
|
|
|
|
## Configuration
|
|
|
|
Centralize configuration loading, processing, precedence, defaults, and
|
|
validation in `internal/core/config`.
|
|
|
|
The goal is to make configuration discoverable and avoid implicit or hidden
|
|
operational values. User-visible defaults and cross-package operational defaults
|
|
should be defined in config code.
|
|
|
|
Unless documented otherwise, precedence is:
|
|
|
|
1. CLI flags
|
|
2. environment variables
|
|
3. configuration file
|
|
4. built-in defaults
|
|
|
|
Prefer YAML configuration unless the project has a strong reason to use another
|
|
format. Config files should be discoverable at
|
|
`/usr/local/etc/notarius/config.yml`, with a CLI override via `--config`.
|
|
|
|
Configuration files should not contain raw secrets unless the application is
|
|
explicitly designed for that. Prefer environment variables or secret files for
|
|
secrets.
|
|
|
|
Adapter-specific and extractor-specific configuration should remain grouped by
|
|
the adapter or extractor that owns it.
|
|
|
|
## Embedded Assets
|
|
|
|
Store embedded JSON schemas, Markdown prompts, templates, and similar assets as
|
|
separate files, not inline string literals, unless there is a strong reason
|
|
otherwise.
|
|
|
|
Embedded prompts and response schemas should have stable IDs, versions, source
|
|
metadata, and hashes suitable for diagnostics and run manifests.
|
|
|
|
## Errors and Logging
|
|
|
|
Errors should be actionable and preserve context. Wrap errors with operation and
|
|
path/resource context. CLI code should convert internal errors into concise
|
|
user-facing messages.
|
|
|
|
Errors and logs must not expose secrets.
|
|
|
|
Use structured logging where practical. Logs should describe operations, paths,
|
|
external calls, retries, and failure causes, but should not include large source
|
|
or artifact payloads by default.
|
|
|
|
## Context, Timeouts, and Cancellation
|
|
|
|
Long-running operations should accept `context.Context`. External calls,
|
|
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
|
|
workflows should respect cancellation and timeouts.
|
|
|
|
## State, Files, and Safety
|
|
|
|
If the application writes durable state, writes should be atomic where
|
|
practical. Multi-step workflows should preserve enough state to support
|
|
inspection, retry, or resume after failure.
|
|
|
|
Code that deletes, moves, or overwrites files must use narrow, explicit paths.
|
|
Avoid broad parent-directory operations. Cleanup that can cause data loss must
|
|
be opt-in.
|
|
|
|
## Testing
|
|
|
|
Core logic should be testable without real external services. Use fakes,
|
|
fixtures, or local test doubles for adapters, extractors, validators, and LLM
|
|
clients where practical.
|
|
|
|
Contract-first work should include fake implementations that prove interfaces
|
|
compose before real adapters or extractors depend on them.
|
|
|
|
Config examples should be load-tested once config files exist. Important CLI
|
|
workflows should have parser or command tests. Adapter, extractor, and validator
|
|
contracts should have focused tests that do not require running the full
|
|
application unless end-to-end coverage is intentional.
|
|
|
|
## Documentation
|
|
|
|
Documentation should follow the project documentation policy. Keep user docs
|
|
focused on implemented behavior. Put future, planned, or aspirational work only
|
|
under `docs/roadmap/`.
|
|
|
|
Core documentation should use generic terms such as source document, source
|
|
unit, source reference, input adapter, extractor, artifact, validator, and run
|
|
manifest.
|
|
|
|
Source-format details belong in adapter or integration docs. Domain-specific
|
|
extraction details belong in extractor or artifact docs.
|
|
|
|
When changing architecture, config, CLI behavior, adapters, extractor contracts,
|
|
validator contracts, LLM runtime behavior, or artifact schemas, update the
|
|
relevant docs and examples in the same change.
|