notarius/docs/roadmap/initial-architecture.md

# Initial Architecture Roadmap

## Status

This document captures proposed architecture and implementation sequencing for
Notarius. It describes planned work, not implemented behavior.

## Goal

Notarius should extract structured JSON artifacts from primary source inputs
using modular, LLM-backed extractors.

The first MVP should target audio transcripts generated by Seriatim. That
choice should be implemented as an input adapter, not as a transcript-specific
assumption in the application core. Later input sources, such as unstructured
Markdown notes or Obsidian documents, should be addable through new adapters and
extractors without reshaping the framework.

The first extraction domain should be D&D session analysis, starting with spell
casts. That domain should live in extractor packages and related schemas, not in
core framework packages.

The application should follow the same broad architecture as Audita:

- deterministic core packages for config, source documents, artifacts, diagnostics, and reporting;
- input adapters that translate external source formats into a small internal source model;
- reusable framework packages for contracts, orchestration, LLM runtime, structured output, and validation;
- independent extractor packages that own domain-specific behavior;
- independent validator packages;
- embedded prompt and JSON schema assets;
- CLI orchestration that wires the pieces together without owning domain logic.

The main domain difference from Audita is that Notarius emits extracted
artifacts rather than proposing and applying transcript corrections.

## Architectural Principles

- Keep the core input model generic: ordered text units plus metadata.
- Keep source-format details in hexagonal input adapters.
- Keep extraction-domain details in extractor packages.
- Treat evidence as source references, not transcript references.
- Prefer narrow, useful abstractions over a universal document model.
- Preserve enough provenance for validation, replay, and downstream inspection.

## Proposed Package Shape

```text
cmd/notarius
internal/cli

internal/core/config
internal/core/source
internal/core/sourcechunking
internal/core/artifacts
internal/core/diagnostics
internal/core/reporting
internal/core/extractorcatalog
internal/core/inputcatalog

internal/adapters/input/seriatim
internal/adapters/input/markdown

internal/framework/contracts
internal/framework/extraction
internal/framework/runner
internal/framework/validators
internal/framework/llm
internal/framework/responseschema
internal/framework/structuredoutput
internal/framework/promptcontext
internal/framework/warnings

internal/extractors/dnd/spells
internal/extractors/dnd/items
internal/extractors/dnd/npcs
internal/extractors/dnd/combat

internal/validators/source_refs
internal/validators/schema_validity
internal/validators/domain_consistency
internal/validators/llm_review

internal/prompts
examples
docs/internal
```

The `markdown` adapter is listed as a likely future package. The MVP should only
implement the Seriatim adapter unless a second adapter is needed to test the
boundary.

## Core Concepts

### SourceDocument

Canonical internal representation of source material. This should be the object
extractors receive, regardless of whether the original input was a transcript,
Markdown file, note export, or another source type.

```go
type SourceDocument struct {
    ID       string         `json:"id"`
    Kind     string         `json:"kind"`
    Format   string         `json:"format"`
    Digest   string         `json:"digest"`
    Units    []SourceUnit   `json:"units"`
    Metadata map[string]any `json:"metadata,omitempty"`
}

type SourceUnit struct {
    ID       string         `json:"id"`
    Kind     string         `json:"kind"`
    Text     string         `json:"text"`
    Metadata map[string]any `json:"metadata,omitempty"`
}
```

Initial source-unit assumptions:

- units are ordered;
- unit IDs are stable within a source document;
- each unit has extractable text;
- adapter-specific metadata may carry speaker, timestamps, heading paths, page
  numbers, or other source details.

### Input Adapter

Hexagonal boundary for external source formats.

```go
type InputAdapter interface {
    Key() string
    Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
}
```

The MVP adapter should target Seriatim minimal transcript JSON. Seriatim segment
fields should map as follows:

- `id` becomes `SourceUnit.ID`;
- `text` becomes `SourceUnit.Text`;
- `speaker`, `start`, and `end` become unit metadata;
- Seriatim output metadata becomes document metadata.

The core runner should not know that these units came from transcript segments.

### SourceRef

Grounding reference from an extracted fact back to source units.

```go
type SourceRef struct {
    SourceID    string `json:"source_id"`
    StartUnitID string `json:"start_unit_id"`
    EndUnitID   string `json:"end_unit_id"`
}
```

Initial source-reference validation should require:

- source ID exists for the current run;
- start and end unit IDs exist;
- start is less than or equal to end in document order;
- the referenced range is contiguous within the source document;
- every extracted fact has at least one source reference unless its schema
  explicitly allows ungrounded metadata.

Transcript-oriented output can still present these as transcript segment ranges
when the adapter metadata makes that interpretation available.

### Extractor

Reusable module contract for producing one artifact type.

```go
type Extractor interface {
    Key() string
    ArtifactType() string
    SchemaVersion() string
    Validators() []Validator
    Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}
```

An extractor should receive either a whole source document or a source slice,
depending on runner configuration. It should return typed artifact candidates
plus warnings. It should not mutate the source document.

Extractor packages own domain concepts. For example, D&D spell extraction should
live under `internal/extractors/dnd/spells`; a future to-do extractor for notes
should live under a different domain path and use the same framework contract.

### Validator

Reusable validation contract for artifact candidates.

Validators should cover:

- JSON/schema validity;
- source-reference validity;
- required-field and shape checks;
- domain consistency;
- optional LLM review for high-risk or ambiguous artifacts.

Validator output should follow Audita's decision-cardinality model: each
candidate artifact receives exactly one decision per validator.

### Artifact

Final approved JSON output from one or more extractors.

Artifacts should preserve enough metadata to support downstream validation,
debugging, and replay. The exact top-level envelope is still open, but should
include artifact type, schema version, extracted records, source references, and
run manifest data.

### RunManifest

Per-run provenance record.

```go
type RunManifest struct {
    InputAdapter     string   `json:"input_adapter"`
    SourceDigests    []string `json:"source_digests"`
    Extractors       []string `json:"extractors"`
    SchemaVersion    string   `json:"schema_version"`
    ValidationStatus string   `json:"validation_status"`
}
```

The manifest should eventually include model names, prompt IDs, prompt hashes,
response schema versions, config source, started/completed timestamps, and
diagnostics paths.

## Initial Extractor Targets

### D&D Spells

Recommended first vertical slice because it is narrow but representative.

```go
type SpellCast struct {
    Player               string      `json:"player"`
    Spell                string      `json:"spell"`
    Effect               string      `json:"effect"`
    NarrativeDescription string      `json:"narrative_description"`
    SourceRefs           []SourceRef `json:"source_refs"`
}
```

The spell extractor should be D&D-specific. The framework should not know what a
spell is.

### D&D Items

Tracks items gained, lost, transferred, consumed, or transformed.

Open questions:

- Should currency be represented as items or as its own artifact type?
- Should item ownership be a required field?
- How should ambiguous ownership changes be represented?

### D&D NPCs

Tracks NPCs interacted with, newly introduced, renamed, described, or otherwise
made relevant to campaign state.

Open questions:

- Should NPC identity resolution happen inside this extractor or in a later
  deduplication stage?
- Should location/faction/relationship facts be separate artifact types?

### D&D Combat

Likely warrants a dedicated schema rather than a generic event list.

Proposed first shape:

```go
type CombatTurn struct {
    Actor                string      `json:"actor"`
    Action               string      `json:"action"`
    Outcome              string      `json:"outcome"`
    NarrativeDescription string      `json:"narrative_description"`
    SourceRefs           []SourceRef `json:"source_refs"`
}
```

Open questions:

- Should combat be extracted as turns, rounds, encounters, or all three?
- Should mechanical fields such as damage, conditions, saves, attacks, and spell
  slots be normalized immediately or added later?
- How should uncertain initiative order be represented?

### Future Non-D&D Extractors

The architecture should support extractors outside the D&D domain. Examples:

- to-do items from Markdown or Obsidian notes;
- decisions and action items from meeting transcripts;
- named people, places, and dates from research notes.

These should be addable as extractor packages without changing runner,
validator, source-reference, or LLM framework contracts.

## Proposed Runner Flow

1. Load effective config.
2. Create diagnostics run directory.
3. Resolve the configured input adapter.
4. Read source input.
5. Parse source input into a `SourceDocument`.
6. Validate source-document invariants.
7. Chunk source units into deterministic source slices.
8. Resolve configured extractor instances through a registry.
9. Execute extractor instances in configured order.
10. Run deterministic validators before LLM-backed validators.
11. Retain approved artifacts and rejected-artifact diagnostics.
12. Merge approved slice artifacts deterministically.
13. Serialize final output JSON.
14. Write run manifest, diagnostics, and optional report JSON.

The runner should operate on source documents and source slices only. Any
transcript-specific behavior should happen before the runner, inside the input
adapter, or after the runner, inside output rendering that understands source
metadata.

## Audita Patterns To Reuse

Reuse these architectural patterns:

- deterministic parsing and schema validation style;
- deterministic chunking of ordered source units;
- explicit extractor registry;
- `contracts` package for transport-neutral interfaces;
- OpenAI-compatible structured LLM client;
- scheduler for bounded LLM concurrency;
- embedded prompt registry with prompt metadata and hashes;
- embedded response-schema registry with schema metadata and hashes;
- diagnostics run directory with redacted effective config;
- validator decision cardinality and deterministic validator ordering;
- CLI tests and fixture-driven integration tests.

Avoid copying these Audita concepts directly:

- transcript-specific core types;
- correction proposals;
- replacement policies;
- deterministic transcript mutation;
- correction ledger terminology.

Those concepts are specific to Audita's transcript-editing role and should be
replaced with source-document, artifact-candidate, artifact-validation, and
extraction-report concepts.

## Checkpoint Roadmap

The initial implementation should proceed through five coherent checkpoints.
Each checkpoint should leave the repository in a reviewable state, with the code
compiling and targeted tests covering the newly introduced contracts or behavior.

1. [Core Contracts And Skeleton](1-core-contracts-and-skeleton.md)
2. [Framework Composition](2-framework-composition.md)
3. [Portable Audita Infrastructure](3-portable-audita-infrastructure.md)
4. [Seriatim Input Adapter](4-seriatim-input-adapter.md)
5. [D&D Spells Extractor](5-dnd-spells-extractor.md)

The first useful vertical slice should arrive at checkpoint 5: Seriatim
transcript input to validated D&D spell artifact output. Earlier checkpoints are
intentionally contract-first and may not produce useful user output yet.

## Open Design Questions

- Should final output be one combined artifact envelope or one file per
  extractor?
- Should extractor output use typed Go structs per artifact or a generic
  artifact record with `json.RawMessage` payloads?
- Should schemas be versioned per extractor, globally, or both?
- Should every record require source references, or should some top-level
  artifact metadata be allowed without source references?
- Should overlapping source-reference ranges be merged, preserved exactly, or
  both?
- Should extraction run independently per source slice only, or should some
  extractors receive whole-document context?
- Should a later reconciliation stage deduplicate entities and events across
  source slices?
- Should LLM review be part of each extractor's validator chain or a separate
  review phase?
- Should the Seriatim adapter accept only its minimal schema initially or also
  support richer transcript schemas?
- Should source-unit metadata be untyped `map[string]any`, typed extension
  structs, or both?

## Near-Term Documentation Tasks

Once behavior is implemented, move implemented contracts out of roadmap docs and
into canonical docs:

- `README.md` for purpose and shortest useful command;
- `docs/cli.md` for CLI behavior;
- `docs/config.md` for config fields and precedence;
- `docs/internal/` for implemented architecture and package boundaries;
- `docs/integrations/` for source input and artifact file formats;
- `examples/` for maintained source, config, and artifact examples.