notarius/docs/roadmap/future.md

# Future Work

Current Notarius behavior is documented in the canonical README, CLI,
configuration, operations, internal, and integration docs. This roadmap records
future work only. Items are ordered roughly by current value and specificity,
not as committed release dates.

## Near-Term Validation And LLM Reliability

The following work forms one related program but should be promoted into
separate feature roadmaps and implemented in dependency order. PromptKit owns
structural output repair within one completion. Notarius owns stage candidates,
validator chains, semantic rejection policy, and whether another stage attempt
is warranted.

### 1. Upgrade To PromptKit v0.8.0

- Upgrade the PromptKit dependency and follow the upstream v0.8.0 release
  guide, including its now-active `repair_attempts` behavior and compatibility
  checks for prompt definitions, prepared execution, validation results,
  capacity accounting, error adaptation, debug data, and cumulative token
  usage.
- Use PromptKit's bounded repair loop for eligible structural output failures.
  A repair must resend the immutable original prompt followed by the latest
  defective assistant response and one bounded user correction message. It
  must not accumulate the history of every defective candidate.
- Review every maintained LLM prompt's structural repair budget. Prefer a
  small positive budget where a corrected structured response can be useful,
  while retaining zero where a second call would be inappropriate or where
  observational evidence does not justify it.
- Keep PromptKit repair attempts distinct from Notarius stage retries.
  PromptKit repairs a response that fails its deterministic `basic`, `json`, or
  `json_schema` contract within one stage attempt; Notarius may later retry a
  complete stage candidate after application validation. Document and test the
  multiplicative maximum provider-call cost created by both configured
  budgets, including PromptKit repair calls made by any LLM-backed validators.
  Attribute generation count, token usage, latency, and failure provenance to
  the producer, validator, PromptKit repair, and Notarius correction layers so
  operators can explain the actual cost of a run.
- Treat exhaustion of PromptKit structural repair as the absence of a usable
  structured candidate. The default terminal policy is to fail the run after
  the applicable Notarius stage-attempt budget is also exhausted. A future
  configurable alternative may reject the affected stage or lane where the
  pipeline can represent that outcome, but it must never accept undecodable or
  structurally invalid output merely with a warning.

### 2. Feedback-Aware Stage Validation Retries

- Model Notarius's corrective stage-retry conversation explicitly after
  PromptKit v0.8.0. The first attempt sends the ordinary complete initial
  prompt. If application validation rejects the resulting LLM-produced
  candidate and another stage attempt is available, reconstruct that complete
  initial prompt byte-for-byte and append exactly two messages: an assistant
  message containing the defective response and an application-owned user
  message detailing every applicable semantic validation error and requesting
  one corrected, complete replacement response. This is a freshly constructed
  correction request, not continuation of an accumulating conversation.
- Use the configured stage `retries` value as the one outer retry budget for
  this loop. `retries: N` continues to mean at most `N` additional complete
  chunk, extract, merge, or normalize attempts after the initial attempt,
  whether an attempt is needed because of a producer error or semantic
  rejection. Do not add a second semantic-correction count. PromptKit's
  prompt-level `repair_attempts` budget is independent and internal to each
  individual LLM completion, and does not consume or replenish the Notarius
  stage budget.
- Extend the framework-managed validation boundary for chunk, extract, merge,
  and normalize stages so a rejected LLM-produced candidate and its exact raw
  model response remain available to construct the next stage attempt.
  Deterministic producers cannot improve by repeating the same inputs; a
  rejection from a deterministic stage is therefore terminal under the
  configured rejection policy rather than consuming retries mechanically.
- Preserve the original session ID, selected profile, structured-output
  contract, prompt inputs, and reusable prompt prefix. Carry only the latest
  candidate and latest aggregate feedback; do not build an unbounded retry
  conversation. Keep model-facing corrective guidance separate from
  operator-facing diagnostics, and apply explicit size, redaction, and debug
  disclosure rules to both.
- Run every applicable validator in the configured chain before deciding
  whether to retry. Do not short-circuit merely because an earlier validator
  rejected the candidate. Aggregate all semantic rejection reason codes and
  corrective guidance into the retry message so one retry can address the
  whole candidate. A validator is applicable only when its declared target and
  prerequisites can be satisfied; record a deterministic skipped diagnostic
  rather than invoking a validator on an input it cannot interpret. Initially
  execute the chain sequentially in configured order so results, diagnostics,
  costs, and feedback ordering remain deterministic; consider validator
  concurrency only in response to measured latency.
- Continue running independent applicable validators after one validator
  execution failure so the attempt retains as much useful diagnostic
  information as practical. Do not present validator operational failures as
  defects in the producer candidate and do not include them in corrective
  feedback.
- Distinguish three terminal conditions and make their policies configurable
  at a coherent pipeline or binding scope:
  - **producer structural failure:** PromptKit could not return a usable
    structured candidate after its repair budget. Default to `fail_run`; an
    allowed alternative may record a terminal stage or lane rejection where
    execution can safely continue, but may not accept the invalid output;
  - **semantic rejection:** one or more validators completed and rejected the
    candidate. Default to `fail_run` after corrective stage retries are
    exhausted; allow an explicit alternative that records the existing
    rejected-output outcome without advancing that output;
  - **validator execution failure:** a validator could not produce a valid
    decision because of generation, structural-output, transport, or internal
    failure. Default to a genuine warning and an explicitly recorded
    `validation_incomplete` or equivalent degraded state while allowing the
    candidate to continue; allow strict configuration to fail the run instead.
- An LLM-backed validator uses the same scheduled PromptKit boundary as every
  other LLM-backed module. Its own response may use PromptKit's bounded
  structural repair. Distinguish its possible output states:
  - output rejected by PromptKit's structural contract should consume only the
    validator prompt's configured PromptKit repair budget;
  - output that is structurally valid but violates a deterministically
    checkable validator-result invariant should be classified as a validator
    execution failure;
  - output that satisfies the complete validator-result contract is the
    validator's decision, even though an LLM judgment may remain imperfect.
    Automatically judging that judgment would require another semantic
    validator and is outside this feature.
  If the validator cannot return a contract-valid decision, do not recursively
  create another Notarius semantic-validation loop around it. Apply the
  configured validator-failure policy. The default warning must identify the
  validator and affected stage without exposing sensitive content.
- Separate validator execution retry from producer correction. A transient
  validator operational failure must not automatically discard and regenerate
  an otherwise usable producer candidate. Any bounded retry of the validator
  itself should reuse that same immutable candidate and remain subordinate to
  PromptKit and provider retry behavior.
- Preserve attempt-level provenance, cumulative token usage, validator
  outcomes, aggregated correction feedback, and terminal policy decisions in
  the debug and manifest models without copying raw source material into
  ordinary errors or durable summaries.
- Define terminal-outcome precedence. A semantic rejection dominates a
  validator execution failure for the same candidate: use the completed
  rejections to correct the producer while separately recording incomplete
  validation. If a later candidate has no semantic rejection but one validator
  still fails, apply the configured validator-failure policy to that candidate.
  Never allow a known semantic rejection to become accepted through a
  warn-and-continue setting, and never accept a structurally invalid producer
  response. Permissive policy may preserve a rejected-output outcome or accept
  a structurally valid candidate with explicitly incomplete validation; it may
  not relabel known-invalid output as approved.

Before implementation, record the generic validation and retry state machine
in an ADR. The ADR should own the separation between PromptKit repair and
Notarius correction, use of the existing stage-retry budget, reconstruction of
correction conversations, all-applicable-validator aggregation, deterministic
validator ordering, non-recursive validator failure handling, outcome
precedence, default fail-open/fail-closed choices, configurable terminal
policies, and provenance and sensitive-data constraints. A dependency-upgrade
ADR is not needed for PromptKit v0.8.0 itself. Current behavior remains
authoritative until the validation ADR is implemented and the canonical
architecture, configuration, operations, and internal documentation are
updated.

### 3. D&D Combat Scene Semantic Validation

- Add an optional production LLM-backed D&D validator that determines whether
  proposed scene boundaries and classifications represent substantive active
  combat correctly. Its central quality goal is that active combat is kept in
  coherent scenes classified as `combat`, rather than split incorrectly or
  hidden inside scenes classified as `narrative`, `recap`, or `meta`.
- Resolve the validator's exact target before implementation. The current
  `dnd/scenes` chunker owns only complete, gap-free source ranges, while the
  per-chunk `dnd/scene-descriptions` extractor owns the `combat`, `narrative`,
  `recap`, and `meta` classification. The preferred initial placement is
  therefore an extract-stage validator for `dnd/scene-descriptions`, where it
  can compare one proposed kind with the corresponding transcript chunk.
- Consider a chunk-stage LLM validator only for a distinct boundary-coherence
  question that can be answered from the complete transcript and proposed
  range map, such as whether one continuous combat was fragmented across
  inappropriate scene boundaries. Do not duplicate the same classification
  judgment at both stages. Moving classification into chunk-plan annotations
  would change the deliberately minimal, annotation-free chunk contract and
  requires an explicit architecture review before it is selected.
- Validate both false negatives and false positives: a non-combat kind must not
  omit substantive active combat, and a combat kind must be supported by such
  combat. Keep the existing deterministic downstream rule that combat-turn
  extraction runs only for an exact `combat` scene classification; semantic
  review improves the upstream classification but does not replace that gate.
- Run the semantic validator through PromptKit, use a minimal required-field
  structured response schema, and let PromptKit repair structural validator
  output within its bounded budget. A contract-invalid final validator response
  is a validator execution failure, not a semantic rejection and not a reason
  to recursively validate the validator.
- Evaluate the prompt and decision policy against a small human-reviewed set
  containing combat setup, active turns, interruptions, multi-phase encounters,
  brief rules discussion, aftermath, recalled combat, and false-positive
  hostile dialogue. Measure false acceptance, false rejection, retry success,
  added calls, latency, and token cost before placing it in the production
  default chain.
- An ADR is not required if classification remains owned by
  `dnd/scene-descriptions` and the validator follows the generic validation ADR.
  Create or supersede an ADR if the work transfers scene classification into
  the chunker or otherwise changes stage ownership or the durable chunk-plan
  contract.

### 4. Warning Signal And Presentation Reform

- Audit every warning producer and representative successful runs. Ordinary
  success producing dozens of warnings is a failed operator experience: the
  volume obscures actionable problems and trains operators to ignore the
  warning channel.
- Define a small warning taxonomy that distinguishes actionable degradation,
  incomplete validation, lossy fallback, and data-quality risk from routine
  normalization observations or informational diagnostics. Preserve detailed
  traceability in debug or manifest data without promoting every observation
  to a top-level CLI warning.
- Consider stable deduplication and aggregation by scope and reason code,
  bounded samples plus omitted counts, and a concise CLI summary with a path to
  detailed diagnostics. Do not suppress genuine validator execution failures
  merely to reduce the count.
- Decide which warnings affect process status, rejection summaries, durable run
  receipts, or only debug output. Ensure warning ordering and aggregation are
  deterministic across concurrent execution.
- Establish a representative warning-volume acceptance target and human review
  workflow before changing individual producers piecemeal. The intended result
  is not zero warnings; it is a small set in which every surfaced warning merits
  operator attention.
- This work does not require an ADR unless it changes validation acceptance,
  failure, or durable contract semantics. CLI presentation and diagnostic
  taxonomy otherwise belong in a feature roadmap followed by updates to their
  canonical configuration, operations, integration, and internal documents.

## Near-Term D&D Pipeline

### Evaluate Spell Extraction And Normalization

- Evaluate ordinary extraction retries and the completed normalization path
  against a human-reviewed transcript set before and after adopting the shared
  PromptKit repair and Notarius validation-retry policies above.
- Maintain a small set of human-reviewed transcripts and outputs for prompt,
  validator, and normalizer development. Treat model-quality review as an
  iterative human evaluation aid, not a deterministic correctness gate.

### Evaluate The Shared D&D Scene Plan

- Reassess whether one shared scene plan provides enough context for NPC,
  spell, combat, interaction, and scene-description lanes after real-world use.
  Add more complex chunking only in response to demonstrated failures.

## Shared Normalization And Quality Work

The implemented source-backed core and initial D&D registry adoption are
described by [Module Internals](../internal/modules.md#semantic-reconciliation)
and
[D&D Module Internals](../internal/dnd.md#semantic-registry-reconciliation).
The [Semantic Reconciliation Roadmap](semantic-reconciliation.md) retains the
original feature scope; the sections below keep broader extensions deferred.

### Large-Collection Semantic Reconciliation

- Evaluate deterministic candidate blocking only after representative registry
  inputs exceed the active roadmap's bounded single-request limits. Blocking
  should use cheap, explainable signals to form plausible comparison sets while
  preserving the possibility that a duplicate appears outside a lexical name
  match.
- Define correctness for candidates that appear in more than one block,
  conflicting canonical selections, transitive identity across blocks, retry
  isolation, and deterministic final ordering before implementation.
- Prefer a reconciliation graph or union plan with explicit conflict checks
  over arbitrary fixed-size slices. Never silently treat a batch boundary as
  evidence that two candidates are distinct.
- Record per-request bounds, block provenance, model calls, discarded
  proposals, and final group derivation well enough to audit a collapse.

### Operator-Selected Semantic Policies

- Consider allowing an operator to select an approved semantic-policy prompt
  for a typed reconciliation module without replacing the shared protocol,
  response schema, or deterministic safety rules.
- Define the trusted asset source, configuration syntax, compatibility checks,
  startup validation, provenance, prompt fingerprinting, checkpoint effects,
  and support boundary before exposing the option.
- Prefer selection among registered, typed-policy-compatible prompt assets over
  arbitrary filesystem prompt paths. Do not add this flexibility until an
  operator workflow requires it; artifact-family-owned policy remains simpler
  and safer for the initial implementation.

### Broader Reconciliation Inputs And Module Selection

- Revisit alternate context providers when a concrete non-source-backed entity
  collection needs semantic reconciliation. Any extension must preserve the
  same request-local identity, deterministic proposal validation, provenance,
  and typed application guarantees.
- Consider a selectable generic normalizer only if Notarius gains a real
  domain-neutral typed artifact contract that can safely support it. Do not
  weaken exact artifact registration or introduce reflection-based arbitrary
  JSON mutation merely to expose a universal module key.

### Validation And Review

- Add domain validators and production default chains alongside each new D&D
  artifact.
- Add production LLM-backed validators only when a concrete review policy
  benefits from model judgment and deterministic checks are insufficient.
- Add validator diagnostics and timing summaries if operators need more detail
  than the current [durable output bundle](../integrations/json-output.md)
  provides.
- Add validator compatibility metadata if deployments need config-time proof
  that a validator is suitable for a particular stage, module, or artifact
  kind.
- Add media-type validators when non-JSON artifact representations are
  introduced.

## Further Reference Evolution

- Make prior-run artifacts easier to bind as references without changing the
  existing module-facing reference-item contract.
- Add structured or parsed references, such as typed NPC registries, rosters,
  or spell catalogs, when opaque UTF-8 prompt material is no longer sufficient.
- Add per-slot or per-chunk inclusion policies so large references are not
  repeated in every prompt unnecessarily.
- Add token budgeting and model context-window management for reference
  content.
- Add reference caching, preprocessing, summarization, embedding, or retrieval
  only when reference size and observed model behavior justify them.
- Extend generated references to prior-run artifacts or derived summaries only
  after same-run ordered handoffs establish the required provenance and
  lifecycle semantics.

## Design Considerations To Revisit

These concerns are relevant to ordered artifact dependencies but are not
committed near-term features.

### Cross-artifact identity links

Evaluate whether downstream D&D artifacts should retain canonical NPC IDs from
the generated NPC reference in addition to normalized display names. Any such
contract must define player-character, unknown-actor, missing-NPC, and
superseded-identity behavior before implementation. Deterministic validation
may confirm that a linked ID exists in the consumed NPC artifact, but the link
must never substitute for transcript evidence that the downstream event
occurred.

### Artifact contract evolution

Define compatibility and migration policy before generated-reference chains
must span multiple schema versions or long-lived historical artifacts. The
policy should address stable identifier semantics, which schema changes permit
checkpoint reuse, when an older artifact may be decoded or adapted, and when a
producer or all dependents must be recomputed. Do not add a general migration
framework until an actual contract change requires one.

### Artifact-family-oriented physical packaging

[ADR-0004](../adr/0004-package-modules-by-domain.md) currently groups production
extensions by domain and then by pipeline stage. After artifact-family
ownership terminology is established and more families span extraction,
normalization, validation, codecs, references, and assets, reassess whether a
feature-first physical layout would improve navigation and reduce scattered
changes enough to justify a repository-wide package migration. Any change must
address Go dependency cycles, registrar ownership, stable public module keys,
and supersession of the affected ADR-0004 decision. Conceptual artifact-family
ownership does not by itself require this move.

## Blue-Sky Platform And Operations

These ideas are intentionally less specified. Promote one into an earlier
section only after a concrete workflow, contract, and priority emerge.

### Platform Extensions

- Additional input adapters, such as Markdown or note-export formats.
- Additional output encoders.
- Concurrent cross-lane entity normalization or broader workflow composition.
- Batching or specialized context-window controls for LLM-backed validators.

### Distribution And Operations

- Packaged release artifacts for alpha distribution.
- Optional generated example-output fixtures with a regeneration procedure.
- Additional diagnostics or reporting views.

### Workspace And Storage

- Default-idempotent run behavior with an explicit force override.
- Remote workspace storage.
- Workspace garbage collection and archival policies.
- Cross-machine checkpoint reuse.