Compare commits
5 Commits
v0.2.0
...
db2adb52da
| Author | SHA1 | Date | |
|---|---|---|---|
| db2adb52da | |||
| fc3c128171 | |||
| 8a15b083a0 | |||
| d9dae2b639 | |||
| 7a4fd7be7a |
12
docs/cli.md
12
docs/cli.md
@@ -40,7 +40,7 @@ pipeline ID and **--input** are required.
|
||||
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
||||
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
||||
| **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
|
||||
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
||||
| **--session-id id** | Override the generated prompt session identifier with a non-empty value for LLM-backed module calls. |
|
||||
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
|
||||
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
|
||||
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
|
||||
@@ -57,6 +57,16 @@ Persistent reasoning settings remain a PromptKit profile concern.
|
||||
**--recompute-step** requires **--resume**; checkpoint requirements and reuse
|
||||
behavior are documented in [Operations](operations.md).
|
||||
|
||||
Every run uses one effective prompt session. Without **--session-id**, Notarius
|
||||
generates a stable `notarius:v1:` identifier from the trimmed resolved input
|
||||
module key and the input file's exact raw bytes. The same module and bytes
|
||||
therefore produce the same identifier, regardless of pipeline, references,
|
||||
profile, retries, or run settings. An explicit non-empty value replaces that
|
||||
default. Session identifiers are visible to providers; they are non-secret
|
||||
correlation identifiers, not credential storage. See
|
||||
[Operations](operations.md#operational-limits) for privacy and workflow
|
||||
guidance.
|
||||
|
||||
### Reference selectors
|
||||
|
||||
Use **--reference** only for a reference slot declared by the selected
|
||||
|
||||
@@ -58,7 +58,7 @@ Built-in defaults are:
|
||||
|
||||
| Field | Default |
|
||||
| --- | --- |
|
||||
| **concurrency.total_llm** | 1 |
|
||||
| **concurrency.total_llm** | 16 |
|
||||
| **concurrency.stage_workers.extract** | Effective **total_llm** |
|
||||
| **output.directory** | **./notarius-output** |
|
||||
| **cache.chunk_plans.mode** | **auto** |
|
||||
|
||||
@@ -27,10 +27,13 @@ notarius run pipeline-id \
|
||||
```
|
||||
|
||||
Use absolute paths for supplied input, configuration, output-root, and
|
||||
reference files. When a stable prompt session identifier or references are
|
||||
needed, pass the supported CLI flags. Supply credentials through Notarius's
|
||||
documented configuration and environment mechanisms, never as command-line
|
||||
arguments or generated secret-bearing configuration.
|
||||
reference files. Notarius generates a stable prompt session for the resolved
|
||||
input module and exact input bytes. Pass **--session-id** only when intentionally
|
||||
grouping different invocations under a different session. Supply credentials
|
||||
through Notarius's documented configuration and environment mechanisms, never
|
||||
as command-line arguments or generated secret-bearing configuration. In
|
||||
particular, a session identifier is provider-visible and is not a credential
|
||||
mechanism.
|
||||
|
||||
Wait for the process before interpreting standard output. Only an exit status
|
||||
of 0 permits decoding the receipt. On a nonzero exit, retain standard error for
|
||||
|
||||
@@ -98,6 +98,11 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
|
||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||
other operational state are not published as bundle files.
|
||||
|
||||
When present, `metadata.session_id` is the effective non-secret routing
|
||||
correlation identifier used for the run. It can be visible to providers and is
|
||||
not a substitute for a cache or checkpoint identity. Its generation and
|
||||
override behavior are defined by the [CLI reference](../cli.md#run).
|
||||
|
||||
Each `llm_profiles` entry identifies effective, non-secret LLM execution
|
||||
provenance:
|
||||
|
||||
|
||||
@@ -48,11 +48,13 @@ adapter boundary. It also retains responsibility for pipeline retries,
|
||||
scheduling, debug persistence, redaction, profile provenance, and conversion
|
||||
from private model responses into durable domain artifacts.
|
||||
|
||||
Notarius sends its trimmed run session through PromptKit's direct session
|
||||
Notarius sends one stable effective session through PromptKit's direct session
|
||||
field, which is authoritative for provider session behavior. It also retains
|
||||
the same value as the `session_id` prompt variable for maintained prompt
|
||||
compatibility. Session IDs are stable, non-secret correlation identifiers and
|
||||
may be exposed to providers and provider observability.
|
||||
compatibility. The generated identifier is 76 ASCII characters, within
|
||||
PromptKit v0.5.0's 256-code-point session limit. Session IDs are non-secret
|
||||
correlation identifiers and may be exposed to providers and provider
|
||||
observability. The CLI contract owns generation and override behavior.
|
||||
|
||||
Notarius records PromptKit's selected backend ID and effective reasoning
|
||||
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||
|
||||
@@ -89,9 +89,11 @@ handoff:
|
||||
profiles;
|
||||
4. materialize external or generated references and record redacted invocation
|
||||
and resolution provenance when debug capture is enabled;
|
||||
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||
requested cache/checkpoint collaborators;
|
||||
6. read the source input and invoke the framework runner; and
|
||||
5. construct registries, the scheduled LLM client, and prepared modules;
|
||||
6. read the source input once, resolve its effective session from the explicit
|
||||
override or resolved input module and raw bytes, then construct requested
|
||||
checkpoint collaborators and invoke the framework runner with that same
|
||||
value; and
|
||||
7. write the runner's logical output files only after a successful run, then
|
||||
complete the command report and user-facing result.
|
||||
|
||||
@@ -102,6 +104,13 @@ final command result. Detailed state lifecycle, resume handling, and physical
|
||||
path confinement are maintained in [Run State Internals](state.md) and
|
||||
[Operations](../operations.md).
|
||||
|
||||
The CLI owns the versioned generated-session policy and resolves the sole
|
||||
effective value before checkpoint construction. It records that value in the
|
||||
final debug invocation summary when capture is enabled and passes it unchanged
|
||||
to checkpoint identity and `pipeline.RunInput`. The public flag and stability
|
||||
contract are defined by the [CLI reference](../cli.md#run); framework and LLM
|
||||
packages only transport the supplied value.
|
||||
|
||||
For `run --json`, the CLI constructs and encodes its private run-result receipt
|
||||
after a successful runner result is available, before it publishes logical
|
||||
output files. It writes the prepared receipt to standard output only after
|
||||
|
||||
@@ -26,9 +26,10 @@ durable schemas. Those responsibilities remain with the module and its
|
||||
|
||||
`PromptKitClient` validates the request target and prompt identity, maps each
|
||||
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||
maps the trimmed request session to PromptKit's direct per-run session field,
|
||||
retains the same value as the `session_id` prompt variable for maintained
|
||||
prompt compatibility, and forwards profile selection. It then creates one
|
||||
passes the supplied request session through to PromptKit's direct per-run
|
||||
session field, retains the same value as the `session_id` prompt variable for
|
||||
maintained prompt compatibility, and forwards profile selection. It does not
|
||||
derive or replace session values; the CLI owns that policy. It then creates one
|
||||
frozen prepared execution, captures its caller-owned credential-redacted
|
||||
details for debug material, and executes that exact snapshot through
|
||||
PromptKit's prepared-execution boundary. The direct field
|
||||
|
||||
@@ -10,11 +10,11 @@ own durable output shapes. Concrete production extensions are covered by
|
||||
## Boundary
|
||||
|
||||
The pipeline framework accepts a resolved composition, registries, shared
|
||||
dependencies, input bytes, and state/debug collaborators. It returns logical
|
||||
output files, normalized artifacts, recorded rejections and warnings, manifest
|
||||
provenance, and checkpoint decisions. The CLI owns process arguments,
|
||||
configuration discovery, physical roots, and placement of returned output
|
||||
files.
|
||||
dependencies, input bytes, a supplied prompt session, and state/debug
|
||||
collaborators. It returns logical output files, normalized artifacts, recorded
|
||||
rejections and warnings, manifest provenance, and checkpoint decisions. The
|
||||
CLI owns process arguments, configuration discovery, session resolution,
|
||||
physical roots, and placement of returned output files.
|
||||
|
||||
The framework has one fixed shape:
|
||||
|
||||
@@ -84,6 +84,10 @@ incompatible producer prevents the consumer step from starting.
|
||||
|
||||
The runner validates its input, installs no-op state collaborators when none
|
||||
were supplied, and serially performs source parsing and chunk-plan selection.
|
||||
It transports the supplied session unchanged to prompt-facing operations and
|
||||
run-manifest metadata; it neither derives a session nor substitutes a parsed
|
||||
source document identifier. The public session contract is owned by the
|
||||
[CLI reference](../cli.md#run).
|
||||
An accepted plan is materialized into source-addressed chunks and passes the
|
||||
configured chunk validators before any lane runs. A chunk rejection is a
|
||||
recorded pipeline outcome: lanes do not start, but the output stage can encode
|
||||
|
||||
@@ -266,16 +266,18 @@ transport-wide cap. Notarius does not add another timeout around PromptKit.
|
||||
The pinned upstream boundary and profile-format links are in
|
||||
[PromptKit Integration](integrations/pkg-promptkit.md).
|
||||
|
||||
Concurrency has two independent layers. Notarius **total_llm** is the
|
||||
application-wide provider-call limit shared by all backends, modules, retries,
|
||||
and validators. PromptKit may impose a narrower admission limit for the
|
||||
selected backend. The effective active-generation bound is the intersection of
|
||||
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
||||
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
||||
PromptKit backend limit and remain bounded by Notarius. For the configured
|
||||
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
|
||||
as a call limit. A positive value makes the effective active local-generation
|
||||
bound the smaller of **total_llm** and that local limit.
|
||||
Concurrency has two independent layers. Notarius **total_llm** defaults to 16
|
||||
and is the application-wide provider-call limit shared by all backends,
|
||||
modules, retries, and validators. PromptKit may impose a narrower admission
|
||||
limit for the selected backend. The effective active-generation bound is the
|
||||
intersection of the Notarius limit, any PromptKit backend limit, and work made
|
||||
available by the pipeline. Built-in OpenRouter profiles use PromptKit's
|
||||
upstream backend limit; endpoint-only profiles have no PromptKit backend limit
|
||||
and remain bounded by Notarius. For the configured local backend, a zero
|
||||
**concurrency_limit** leaves only the Notarius scheduler as a call limit. A
|
||||
positive value makes the effective active local-generation bound the smaller
|
||||
of **total_llm** and that local limit, so a local limit of four permits no more
|
||||
than four active local generations.
|
||||
|
||||
For a positive local limit, PromptKit owns its default waiting capacity and
|
||||
admission behavior. When a PromptKit backend has admitted all active and queued
|
||||
@@ -289,3 +291,11 @@ under [PromptKit profiles](config.md#promptkit-profiles) and
|
||||
limits and actual provider-call limits are independent. Notarius writes local
|
||||
filesystem state only; remote storage, archival, and retention automation are
|
||||
outside the implemented CLI.
|
||||
|
||||
Every run has an effective prompt session used for provider routing and run
|
||||
provenance. The generated default is stable for the same input module and raw
|
||||
input bytes; use [**--session-id**](cli.md#run) only when intentionally grouping
|
||||
different invocations. Both generated and explicit values can be visible to
|
||||
providers, manifests, checkpoints, and requested debug bundles. Do not put
|
||||
credentials or other secrets in an explicit session identifier; command-line
|
||||
values are not a credential mechanism.
|
||||
|
||||
@@ -53,71 +53,6 @@ not as committed release dates.
|
||||
spell, combat, interaction, and scene-description lanes after real-world use.
|
||||
Add more complex chunking only in response to demonstrated failures.
|
||||
|
||||
## Cross-Cutting LLM Runtime
|
||||
|
||||
### Deterministic Prompt Session Identity
|
||||
|
||||
- Replace the source-document-ID default for prompt sessions with one
|
||||
predictable, procedurally generated session ID for the complete
|
||||
source-processing workload.
|
||||
- Preserve an explicit non-empty `--session-id` as the highest-precedence
|
||||
override. Otherwise, derive the default only from the effective input module
|
||||
identity and the exact raw input bytes.
|
||||
- Use a versioned, bounded representation such as
|
||||
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
|
||||
must fit PromptKit's session length contract and must not embed source
|
||||
content.
|
||||
- Keep the derived session stable across runs, pipelines, selected lanes,
|
||||
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
|
||||
overrides, and output, debug, or cache settings.
|
||||
- Do not include file-backed references, generated references, reference
|
||||
contents, or the composition of a reference bundle in session derivation.
|
||||
References may change between prompt calls within one pipeline without
|
||||
changing routing affinity.
|
||||
- Resolve the authoritative session before checkpoint construction and use the
|
||||
same value for checkpoint runtime identity, every prompt-facing module,
|
||||
PromptKit's direct session field, the compatibility `session_id` prompt
|
||||
variable, run-manifest metadata, and debug metadata.
|
||||
- Keep routing identity separate from cache and checkpoint content identity.
|
||||
Exact prompt prefixes, reference contents, model settings, and other
|
||||
generation-affecting inputs must continue to participate in their existing
|
||||
hashes and checkpoint fingerprints even though they do not change the
|
||||
session.
|
||||
- Treat the generated value as a provider-visible, stable pseudonymous
|
||||
correlation identifier. Do not introduce an installation-specific HMAC or
|
||||
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||
sacrificing deterministic identity across installations.
|
||||
|
||||
### Raise The Default Application-Wide LLM Limit
|
||||
|
||||
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||
single-backend runs can use PromptKit's expected OpenRouter capacity and
|
||||
lower-capacity local backends without an unnecessarily narrower Notarius
|
||||
limit.
|
||||
- Keep the Notarius application-wide scheduler mandatory and require
|
||||
`total_llm` to remain a positive integer. Do not make the default unlimited:
|
||||
endpoint-only profiles, an unrestricted local backend, injected clients, and
|
||||
aggregate work across several backends may have no narrower PromptKit limit.
|
||||
- Continue defaulting `concurrency.stage_workers.extract` to the effective
|
||||
`total_llm`, making its default 16 as part of the same change. Preserve an
|
||||
explicit lower extract-worker setting when an operator wants less queued or
|
||||
concurrent extraction work.
|
||||
- Define effective provider concurrency as the intersection of the Notarius
|
||||
application-wide limit, the selected PromptKit backend limit when present,
|
||||
and the work made available by stage execution. A Notarius limit of 16 does
|
||||
not narrow a backend already limited to 16, while a local backend limited to
|
||||
4 remains bounded at 4.
|
||||
- Treat the default as an application-wide safety ceiling across profiles,
|
||||
backends, modules, retries, and validators. A run that intentionally needs
|
||||
the combined capacity of several backends may configure a higher
|
||||
`total_llm` and an appropriate extract-worker count explicitly.
|
||||
- Retain the existing configuration and environment override surfaces. Update
|
||||
canonical configuration, operations, and internal documentation together
|
||||
when the default changes.
|
||||
- Reconsider decoupling the extract-worker default from `total_llm` only after
|
||||
mixed-backend workloads demonstrate a need for a high global emergency
|
||||
ceiling with a lower default work-production rate.
|
||||
|
||||
## Shared Normalization And Quality Work
|
||||
|
||||
### Generic LLM-Assisted Deduplication
|
||||
|
||||
@@ -1,586 +0,0 @@
|
||||
# PromptKit v0.5 Implementation Plan
|
||||
|
||||
## Objective
|
||||
|
||||
Implement the target state in
|
||||
[PromptKit v0.5 Integration And LLM Profile Policy](promptkit.md). Each numbered
|
||||
stage is intended to be one implementation prompt for a GPT-5.6-Terra coding
|
||||
agent. Complete stages in order and leave the repository buildable, tested, and
|
||||
internally coherent after every stage.
|
||||
|
||||
Follow [Architecture](../policy/architecture.md),
|
||||
[Testing Policy](../policy/testing.md), and
|
||||
[Documentation Policy](../policy/documentation.md) throughout. Preserve
|
||||
unrelated user changes. Use `apply_patch` for source and documentation edits,
|
||||
run `gofmt` on changed Go files, and add only tests that protect the behaviors
|
||||
and risks assigned to that stage.
|
||||
|
||||
Do not implement the separate deterministic session-ID or default-concurrency
|
||||
roadmap items as part of this plan. Do not perform paid or credentialed LLM
|
||||
calls.
|
||||
|
||||
## Background Summary
|
||||
|
||||
Notarius currently pins PromptKit v0.3.0, calls `Prepare` and then `Run` for one
|
||||
completion, validates profiles through a synthetic prompt, has no application
|
||||
fallback profile source, and accepts LLM profiles only at individual bindings
|
||||
or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible
|
||||
with the current tree; a temporary v0.5.0 module override has already passed
|
||||
`go test ./...`.
|
||||
|
||||
The implementation must nevertheless treat the upstream optional-parameter
|
||||
change as intentional: unset `temperature`, `max_tokens`, and `top_p` remain
|
||||
unset and are omitted from compatible provider requests. Do not restore the old
|
||||
implicit `top_p: 1` default.
|
||||
|
||||
## Stage 1: Upgrade The PromptKit Dependency
|
||||
|
||||
### Goal
|
||||
|
||||
Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.
|
||||
|
||||
### Work
|
||||
|
||||
- Update `go.mod` and `go.sum` from PromptKit v0.3.0 to v0.5.0 and run
|
||||
`go mod tidy`.
|
||||
- Change the PromptKit built-in profile-catalog marker in
|
||||
`internal/framework/llm/promptkit_profile_fingerprint.go` to identify
|
||||
v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior
|
||||
catalog identity.
|
||||
- Review PromptKit-facing compile errors or test failures against the v0.4.0
|
||||
and v0.5.0 release guides. Do not adopt prepared execution, inspection, or
|
||||
fallback profiles in this stage.
|
||||
- Replace the existing test assertion for one exact built-in fingerprint hash
|
||||
with durable assertions that the fingerprint is deterministic, non-empty,
|
||||
non-secret, and changes when a semantic profile source changes. Do not add a
|
||||
new version-constant or exact-hash change detector.
|
||||
- Update `docs/integrations/pkg-promptkit.md` to pin and link v0.5.0 and state
|
||||
the implemented dependency-level behavior: unset optional sampling controls
|
||||
are provider defaults. Do not document later stages as implemented.
|
||||
- Update any other canonical text that explicitly claims the dependency is
|
||||
v0.3.0, but defer descriptions of unimplemented v0.5 APIs.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- `go test ./internal/framework/llm ./internal/cli`
|
||||
- `go test ./...`
|
||||
- `go vet ./...`
|
||||
- `go build ./cmd/notarius`
|
||||
- `rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .`
|
||||
- `git diff --check`
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The repository directly pins v0.5.0 and all default offline checks pass.
|
||||
- The profile-source fingerprint identifies the new upstream catalog without a
|
||||
brittle literal-hash test.
|
||||
- Current documentation no longer identifies v0.3.0 as the supported version.
|
||||
|
||||
## Stage 2: Execute One Frozen Prepared Snapshot
|
||||
|
||||
### Goal
|
||||
|
||||
Make Notarius debug details and generation use one exact PromptKit preparation.
|
||||
|
||||
### Work
|
||||
|
||||
- Refactor `PromptKitClient.CompleteStructured` to call
|
||||
`PrepareExecution`, immediately defer `Discard`, obtain a caller-owned
|
||||
`Details` value, and execute with `RunPrepared`.
|
||||
- Preserve the existing Notarius request mapping, cancellation precedence,
|
||||
validation classification, raw structured bytes, response decoding,
|
||||
profile recording, usage reporting, and credential redaction.
|
||||
- Ensure every preparation, execution, validation, empty-result, and decode
|
||||
error retains useful Notarius prompt context without exposing prepared handle
|
||||
state or secrets.
|
||||
- Use `errors.As` to obtain `*promptkit.CapacityError` on admission rejection.
|
||||
Preserve `contracts.ErrLLMCapacityExceeded` as the stable classification and
|
||||
add a nonblank backend ID only to safe application-owned diagnostic context.
|
||||
Do not expose `promptkit.CapacityError` outside the LLM adapter.
|
||||
- Update `docs/internal/llm.md` and the implemented-mechanics portion of
|
||||
`docs/integrations/pkg-promptkit.md` to describe the single frozen execution
|
||||
snapshot and structured capacity adaptation.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Adapt existing PromptKit client tests to the prepared-execution path.
|
||||
- Retain or add one behavioral test proving that the debug prompt details match
|
||||
the request actually passed to generation when a backing prompt source could
|
||||
otherwise change between independent preparations. Test the resulting
|
||||
snapshot consistency, not a private helper call count.
|
||||
- Retain capacity tests proving `errors.Is` reaches
|
||||
`contracts.ErrLLMCapacityExceeded`, the selected backend can appear in safe
|
||||
diagnostic context, and provider calls are not made after rejected
|
||||
admission.
|
||||
- Run `go test ./internal/framework/llm` and
|
||||
`go test -race ./internal/framework/llm`.
|
||||
- Run `go test ./...` and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- `CompleteStructured` no longer calls independent `Prepare` and `Run`
|
||||
operations for one request.
|
||||
- Debug prompt material and generation result originate from the same frozen
|
||||
PromptKit snapshot.
|
||||
- Capacity remains a provider-neutral Notarius error classification.
|
||||
|
||||
## Stage 3: Replace Synthetic Profile Validation With Inspection
|
||||
|
||||
### Goal
|
||||
|
||||
Validate profiles through PromptKit's exact profile-inspection boundary and
|
||||
centralize engine profile-source construction.
|
||||
|
||||
### Work
|
||||
|
||||
- Introduce a small provider-adapter-owned profile inspection or validation
|
||||
function in `internal/framework/llm`. Its public internal signature must use
|
||||
Notarius-owned configuration and result/error types rather than returning
|
||||
PromptKit types to the CLI.
|
||||
- Share the code that applies `profile_dir`, `profile_file`, and registered
|
||||
backend options between the production PromptKit engine and the inspection
|
||||
engine. Preserve the mutual-exclusion and local-backend rules.
|
||||
- Change CLI explicit-profile preflight to use `Engine.InspectProfile` through
|
||||
that LLM boundary.
|
||||
- Remove `profileCheckPromptID`, `profileCheckPromptFS`, the `testing/fstest`
|
||||
production dependency, and the synthetic `Prepare` request.
|
||||
- Preserve distinct, useful errors for an absent profile, invalid profile,
|
||||
unknown backend registration, cancellation, and invalid profile source.
|
||||
- Do not require `api_key_env` to be populated during configuration validation.
|
||||
Inspection may report credential requirements internally, but actual
|
||||
preparation remains responsible for credential availability before a model
|
||||
call.
|
||||
- Update current-behavior sections in `docs/internal/cli.md` and
|
||||
`docs/internal/llm.md`. Keep field definitions in `docs/config.md`.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Replace synthetic-prompt tests with profile inspection tests covering:
|
||||
configured local backend success; missing local backend failure; absent
|
||||
profile; malformed profile; and an otherwise valid profile whose credential
|
||||
environment variable is intentionally unset.
|
||||
- Prove validation performs no provider HTTP call and remains offline.
|
||||
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- No production synthetic profile-check prompt remains.
|
||||
- Profile validation uses the same ordinary profile source and backend
|
||||
registrations as execution.
|
||||
- Configuration validation succeeds for structurally valid profiles without
|
||||
reading credential values.
|
||||
|
||||
## Stage 4: Add Application Fallback Profile Asset Plumbing
|
||||
|
||||
### Goal
|
||||
|
||||
Allow module families to register application-owned fallback profile YAML
|
||||
without placing domain policy in generic LLM code.
|
||||
|
||||
### Work
|
||||
|
||||
- Extend `internal/framework/llm.AssetRegistry` with a separate fallback
|
||||
profile source collection, registration method, flattened filesystem, and
|
||||
safe content digest.
|
||||
- Reuse the existing asset-source path validation and flattening behavior where
|
||||
appropriate. Reject invalid roots, unreadable assets, and duplicate flattened
|
||||
paths. Do not parse PromptKit profile YAML in Notarius.
|
||||
- Add `promptkit.WithFallbackProfileFS` to production engine options only when
|
||||
at least one fallback profile source is registered.
|
||||
- Supply the identical assembled fallback source to the profile-inspection
|
||||
engine. Adjust CLI composition so pipeline-aware profile validation can use
|
||||
the production LLM asset registry without exposing PromptKit types.
|
||||
- Extend profile-source checkpoint identity to include the exact fallback
|
||||
profile asset digest in addition to the PromptKit catalog marker and operator
|
||||
source. Keep the resulting fingerprint hash-only and path/content/credential
|
||||
free.
|
||||
- Keep operator source precedence owned by PromptKit. Do not implement profile
|
||||
merging or duplicate PromptKit source resolution in Notarius.
|
||||
- Update `docs/internal/llm.md` only for the new implemented generic asset and
|
||||
fingerprint mechanics. No domain fallback exists until Stage 5.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add focused AssetRegistry tests for successful flattening, invalid roots,
|
||||
duplicate paths, and hash changes when fallback bytes change.
|
||||
- Add adapter-level tests showing that the fallback filesystem reaches both
|
||||
execution construction and inspection construction.
|
||||
- Extend checkpoint tests to prove fallback content changes profile-source
|
||||
identity without exposing raw YAML or paths. Use relational comparisons, not
|
||||
a fixed hash literal.
|
||||
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Generic plumbing can carry application fallback profiles while remaining
|
||||
unaware of D&D IDs or model settings.
|
||||
- Inspection, execution, and checkpoint identity use the same fallback asset
|
||||
source.
|
||||
|
||||
## Stage 5: Adopt The D&D `dnd-extraction` Fallback
|
||||
|
||||
### Goal
|
||||
|
||||
Give the D&D module family one stable embedded workload profile that operators
|
||||
can replace.
|
||||
|
||||
### Work
|
||||
|
||||
- Add a D&D-owned embedded PromptKit profile asset with ID `dnd-extraction`
|
||||
under `internal/modules/dnd`. Use the exact baseline defined in
|
||||
`promptkit.md`: OpenRouter, `openai/gpt-5.6-luna`, no explicit reasoning
|
||||
effort, a 240-second timeout, flex service tier, and no selected temperature,
|
||||
token limit, or `top_p`. The omitted reasoning value intentionally allows
|
||||
OpenAI's backend to apply its `medium` default.
|
||||
- Register the profile filesystem from the D&D registrar through the generic
|
||||
fallback profile asset boundary. Keep D&D policy out of
|
||||
`internal/framework/llm` and the CLI composition root.
|
||||
- Change every maintained D&D LLM prompt definition—including scene chunking,
|
||||
all D&D extractors, and NPC normalization—from the model-named default to
|
||||
`default_profile: dnd-extraction`.
|
||||
- Add an integration-level profile-resolution test proving that:
|
||||
- the fallback resolves when no operator source defines the ID;
|
||||
- a valid operator profile with the same ID wins completely; and
|
||||
- an invalid matching operator profile fails rather than falling through.
|
||||
- Test through Notarius's assembled production assets and PromptKit boundary;
|
||||
do not duplicate every upstream source-precedence case.
|
||||
- Update the implemented profile ownership and prompt-default behavior in
|
||||
`docs/internal/dnd.md`, `docs/internal/llm.md`, and
|
||||
`docs/integrations/pkg-promptkit.md`. Defer the complete operator walkthrough
|
||||
and examples to Stage 10.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Run focused D&D prompt preparation tests and the production composition
|
||||
tests.
|
||||
- Run `go test ./internal/modules/dnd/... ./internal/framework/llm
|
||||
./internal/cli`.
|
||||
- Run `go test ./...`.
|
||||
- Verify `rg -n 'default_profile: gemini-2-flash' internal/modules/dnd`
|
||||
returns no matches.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- All maintained D&D prompts use the application-owned logical profile ID.
|
||||
- The fallback works without an operator profile and remains authoritatively
|
||||
overridable by a matching valid operator definition.
|
||||
|
||||
## Stage 6: Introduce Module Execution-Class Metadata
|
||||
|
||||
### Goal
|
||||
|
||||
Make each production module's ability to use an LLM statically discoverable
|
||||
without yet changing profile inheritance.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `ExecutionClass contracts.ExecutionClass` to `pipeline.ModuleSpec` and
|
||||
preserve it through normalization, cloning, catalogs, registries, JSON/debug
|
||||
views, and lookup helpers.
|
||||
- In this transitional stage only, allow an omitted execution class to
|
||||
normalize to deterministic so existing test-only fixtures can be migrated in
|
||||
Stage 7 without breaking the repository midway.
|
||||
- Explicitly classify every production module:
|
||||
- D&D scene chunking, every D&D extractor, and D&D NPC normalization as
|
||||
`llm_backed`;
|
||||
- all other current production input, chunk, merge, normalize, and output
|
||||
modules as `deterministic`.
|
||||
- Update production module specification tests and production catalog tests to
|
||||
assert the semantic class alongside stage, artifact kind, and capabilities.
|
||||
- Add catalog lookup support needed by later resolution to retrieve a selected
|
||||
module's execution class by stage and key without constructing it.
|
||||
- Do not implement pipeline-level profile inheritance or reject deterministic
|
||||
profiles yet.
|
||||
- Update `docs/internal/modules.md` and `docs/internal/dnd.md` to identify
|
||||
execution class as registered module metadata, while noting only implemented
|
||||
uses.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Run module registration/spec tests across generic, Seriatim, and D&D
|
||||
families.
|
||||
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||
- Run `go test ./...` and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Every production module has an explicit correct execution class.
|
||||
- Catalog consumers can retrieve that class without a concrete module
|
||||
instance.
|
||||
- Test-only omitted classes remain the only temporary compatibility behavior.
|
||||
|
||||
## Stage 7: Enforce Execution Metadata And Remove Runtime Probing
|
||||
|
||||
### Goal
|
||||
|
||||
Finish the execution-class contract so missing metadata cannot cause future
|
||||
profile drift.
|
||||
|
||||
### Work
|
||||
|
||||
- Update every framework, CLI, and integration test module specification to
|
||||
declare an explicit execution class appropriate to the fake behavior.
|
||||
- Change module-spec validation so an empty or unsupported execution class is a
|
||||
registration error. Remove the transitional deterministic default from
|
||||
Stage 6.
|
||||
- Replace the chunk runner's special `ChunkExecutionClassProvider` probe with
|
||||
specification-derived behavior. Remove the now-redundant provider interface,
|
||||
implementation methods, and tests when they have no remaining consumer.
|
||||
- Ensure chunk producer provenance remains unchanged: it records a non-empty
|
||||
effective binding profile for an LLM-backed chunker, while a deterministic
|
||||
chunker records no profile. A profile selected only through the prompt
|
||||
default remains represented by PromptKit's actual-profile manifest rather
|
||||
than being invented as an explicit chunk binding.
|
||||
- Review helper constructors and fixtures for opportunities to set execution
|
||||
class once without obscuring the class under test. Do not introduce an
|
||||
elaborate test-spec framework.
|
||||
- Update internal documentation if the removal changes any described runtime
|
||||
mechanics.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add or retain focused registration tests for missing and invalid execution
|
||||
classes.
|
||||
- Retain chunk-plan provenance tests for LLM-backed and deterministic
|
||||
chunkers.
|
||||
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- No registered module specification relies on an implicit execution class.
|
||||
- Pipeline metadata, not a concrete runtime type assertion, owns module
|
||||
execution classification.
|
||||
|
||||
## Stage 8: Resolve Programmatic Pipeline Profile Defaults
|
||||
|
||||
### Goal
|
||||
|
||||
Implement profile inheritance and precedence inside the pipeline resolver
|
||||
before exposing the field through YAML configuration.
|
||||
|
||||
### Work
|
||||
|
||||
- Add an optional trimmed `LLMProfile` field to
|
||||
`pipeline.PipelineProfile`. Add a non-empty runtime override field to
|
||||
`pipeline.ResolveOptions` so all precedence decisions occur in the resolver
|
||||
rather than through pre-resolution mutation.
|
||||
- After module selection, `--only` filtering, default validator-chain
|
||||
selection, and validator compatibility resolution, apply effective profiles
|
||||
to every selected input, chunk, extract, merge, normalize, output, and
|
||||
validator binding according to the precedence in `promptkit.md`.
|
||||
- Apply profiles only when the selected module or validator execution class is
|
||||
`llm_backed`.
|
||||
- Reject a binding-specific `llm_profile` on any deterministic module or
|
||||
validator. Do not reject or inspect an unused pipeline default when no
|
||||
selected LLM-backed binding consumes it.
|
||||
- Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is
|
||||
selected so PromptKit can use the prompt's `default_profile`.
|
||||
- Store the effective values on resolved bindings before digest construction.
|
||||
Do not add a second inheritance decision to execution.
|
||||
- Ensure semantically equivalent repeated binding profiles and one inherited
|
||||
default produce the same resolved pipeline digest. Ensure any changed
|
||||
effective profile changes the digest.
|
||||
- Do not modify file configuration or CLI parsing in this stage.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add pipeline package tests for the complete precedence matrix:
|
||||
runtime override; binding-specific exception; pipeline default; prompt
|
||||
fallback; and deterministic bindings.
|
||||
- Cover default and explicitly configured validator chains, all relevant stage
|
||||
categories, `--only` lane selection, unused defaults, deterministic-profile
|
||||
rejection, and semantic digest equivalence.
|
||||
- Prefer table-driven package-level tests over assertions on private traversal
|
||||
helpers.
|
||||
- Run `go test ./internal/framework/pipeline` and `go test ./...`.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Programmatic pipelines resolve one canonical effective profile policy.
|
||||
- Only LLM-backed resolved bindings can contain a profile.
|
||||
- Runtime override, binding, pipeline, and prompt precedence is unambiguous and
|
||||
digest-stable.
|
||||
|
||||
## Stage 9: Expose Pipeline Defaults Through Configuration And CLI
|
||||
|
||||
### Goal
|
||||
|
||||
Make the profile-default workflow available to operators while preserving
|
||||
validation and override behavior.
|
||||
|
||||
### Work
|
||||
|
||||
- Add optional `pipelines.<id>.llm_profile` support to the version 4 file
|
||||
configuration model. Use presence-aware decoding so an explicitly set blank
|
||||
value is rejected, while omission remains valid.
|
||||
- Preserve the field through file application, configuration cloning,
|
||||
effective configuration, and programmatic profile copies without aliasing or
|
||||
trimming drift.
|
||||
- Remove `applyLLMProfileOverride`. Pass the CLI override through the resolver's
|
||||
runtime-override input so deterministic bindings are never populated.
|
||||
- Update effective profile-ID collection to cover every selected LLM-backed
|
||||
module stage and LLM-backed validator, including future LLM-backed input and
|
||||
output modules. Do not inspect deterministic or unselected profiles.
|
||||
- Ensure `run`, `config validate --pipeline`, resume/checkpoint identity, and
|
||||
relevant dry preflight paths all use the same resolved effective profiles.
|
||||
- Preserve `--llm-profile` as the highest-precedence non-empty run-wide
|
||||
override and preserve binding-specific profiles as exceptions when no CLI
|
||||
override is present.
|
||||
- Do not increment the configuration version.
|
||||
- Update current configuration and CLI contracts in `docs/config.md` and
|
||||
`docs/cli.md` in the same stage. Link to operations for the deployment
|
||||
workflow rather than duplicating it prematurely.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add file-config tests for omission, trimming, explicit blank rejection,
|
||||
unknown-key behavior, cloning, and round-trip application.
|
||||
- Add effective-config and CLI contract tests for precedence, LLM-only
|
||||
application, inherited-profile inspection failure before factory execution,
|
||||
`--only`, and digest changes.
|
||||
- Retain offline operation and do not require credentials for
|
||||
`config validate --pipeline`.
|
||||
- Run `go test ./internal/core/config ./internal/framework/pipeline
|
||||
./internal/cli`.
|
||||
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Operators can select `dnd-extraction` once per pipeline.
|
||||
- Configuration and CLI paths share the resolver's precedence policy.
|
||||
- Unknown effective profiles fail preflight, while deterministic and unused
|
||||
profiles do not cause spurious inspection.
|
||||
|
||||
## Stage 10: Complete Operator Documentation, Examples, And Decision Record
|
||||
|
||||
### Goal
|
||||
|
||||
Make the implemented workflow understandable, copyable, and maintainable
|
||||
without duplicating canonical facts.
|
||||
|
||||
### Work
|
||||
|
||||
- Create an ADR using the next sequential number for the durable decision to
|
||||
use workload-oriented pipeline defaults with operator-overridable application
|
||||
fallback profiles. Record context, decision, alternatives, and consequences;
|
||||
do not turn the ADR into a field reference or implementation log.
|
||||
- Complete `docs/config.md` as the canonical owner of profile-source fields,
|
||||
`pipelines.<id>.llm_profile`, validation, and precedence.
|
||||
- Complete `docs/operations.md` with an operator workflow that distinguishes
|
||||
Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins,
|
||||
and deployment filesystem profiles. Include production/development/local use
|
||||
of the same `dnd-extraction` ID, credential handling, absolute-path guidance,
|
||||
and the fact that current relative profile paths use the process working
|
||||
directory rather than the configuration file's directory.
|
||||
- Complete `docs/integrations/pkg-promptkit.md` with the v0.5.0 boundary,
|
||||
prepared execution, inspection, fallback and ordinary source precedence,
|
||||
optional provider controls, capacity adaptation, and compatibility policy.
|
||||
- Update `docs/internal/configuration.md`, `docs/internal/pipeline.md`,
|
||||
`docs/internal/cli.md`, `docs/internal/llm.md`, `docs/internal/modules.md`, and
|
||||
`docs/internal/dnd.md` only for their owned implementation details. Link to
|
||||
canonical configuration, operations, and upstream format contracts rather
|
||||
than restating them.
|
||||
- Keep exactly the existing two D&D configuration examples. Add
|
||||
`llm_profile: dnd-extraction` to the minimal and complete pipelines and remove
|
||||
the now-redundant model-named binding override from the complete example.
|
||||
- Add one secret-free maintained operator profile at
|
||||
`examples/profiles/dnd-extraction.yml`. It should be a complete valid profile
|
||||
for the same logical ID and may mirror the embedded baseline; its purpose is
|
||||
to demonstrate file ownership and format, not claim automatic environment
|
||||
detection. Link it from the configuration and operations documentation.
|
||||
- If the complete example selects the external profile file, use a path that
|
||||
is valid for the documented repository-root invocation and explicitly note
|
||||
the working-directory rule. Keep the minimal example dependent only on the
|
||||
embedded fallback.
|
||||
- Add or extend maintained-example validation so both configuration examples
|
||||
and the profile YAML are checked without generation or credentials.
|
||||
- Remove the now-implemented `Pipeline-Level LLM Profile Defaults` section from
|
||||
`docs/roadmap/future.md`. Preserve the unrelated deterministic session and
|
||||
concurrency items.
|
||||
- Do not delete `promptkit.md` or this implementation plan during the feature
|
||||
implementation; retire them only after post-implementation review.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Run maintained example/configuration tests and relevant CLI help/parser
|
||||
tests.
|
||||
- Run `go test ./...`.
|
||||
- Run `rg -n 'gemini-2-flash' examples docs` and review every remaining match
|
||||
for intentional model-policy or historical context.
|
||||
- Run `rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' .` and resolve
|
||||
stale production or current-documentation matches.
|
||||
- Verify all new links and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Every current fact has one canonical documentation owner.
|
||||
- Operators can distinguish and deploy all profile layers without reading Go
|
||||
source.
|
||||
- Both maintained configurations and the maintained external profile are valid,
|
||||
secret-free, and tested offline.
|
||||
- Implemented profile work no longer remains in `future.md`.
|
||||
|
||||
## Stage 11: Final Verification And Quality Review
|
||||
|
||||
### Goal
|
||||
|
||||
Verify the complete migration as one integrated change and correct only defects
|
||||
or omissions found during that review.
|
||||
|
||||
### Work
|
||||
|
||||
- Review the final diff against every acceptance criterion in `promptkit.md`.
|
||||
- Confirm provider-specific PromptKit types remain inside the LLM integration
|
||||
boundary and D&D policy remains inside the D&D module family.
|
||||
- Confirm execution and inspection receive identical ordinary, fallback, and
|
||||
backend configuration.
|
||||
- Confirm no paths, profile YAML, endpoints, credentials, or prepared handle
|
||||
state leak into fingerprints or ordinary diagnostics.
|
||||
- Confirm all production module specs have explicit correct execution classes
|
||||
and every resolved deterministic binding is profile-free.
|
||||
- Confirm prompt default, pipeline default, binding override, and CLI override
|
||||
behavior through representative assembled configurations.
|
||||
- Review tests for redundancy and remove obsolete synthetic-prompt,
|
||||
runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by
|
||||
stronger contract tests.
|
||||
- Perform an optional manual D&D quality comparison if credentials and an
|
||||
evaluation transcript are deliberately supplied. Record no private input or
|
||||
credential material, and do not make this comparison a completion gate.
|
||||
|
||||
### Validation Commands
|
||||
|
||||
```sh
|
||||
gofmt -w <changed-go-files>
|
||||
go test ./...
|
||||
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Also run focused stale-contract searches:
|
||||
|
||||
```sh
|
||||
rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
|
||||
rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples
|
||||
```
|
||||
|
||||
Review any matches rather than deleting intentional historical references
|
||||
blindly.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- All automated checks pass offline and without real credentials.
|
||||
- The implemented behavior matches `promptkit.md` with no known architecture,
|
||||
provenance, checkpoint, profile-precedence, or documentation gap.
|
||||
- Any optional live evaluation is clearly separate from correctness testing.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The roadmap decisions are sufficient to implement every stage without an
|
||||
additional product or architecture choice.
|
||||
@@ -1,318 +0,0 @@
|
||||
# PromptKit v0.5 Integration And LLM Profile Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the target state for upgrading Notarius from PromptKit
|
||||
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
|
||||
directly improve Notarius. It also defines the application policy for stable,
|
||||
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
|
||||
profile validation, provider defaults, checkpoint identity, and documentation.
|
||||
|
||||
The ordered work needed to reach this state belongs in
|
||||
[the implementation plan](implementation.md). Current behavior remains defined
|
||||
by the canonical documentation outside `docs/roadmap/` until the corresponding
|
||||
work is implemented.
|
||||
|
||||
## Background
|
||||
|
||||
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
|
||||
for debug material and then independently runs the original request, causing
|
||||
PromptKit to prepare the same logical call a second time. The CLI validates an
|
||||
explicit profile by preparing a synthetic prompt. PromptKit profile selection
|
||||
can be repeated on individual module bindings or replaced for one invocation
|
||||
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
|
||||
profile policy.
|
||||
|
||||
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
|
||||
areas:
|
||||
|
||||
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
|
||||
adds opaque prepared executions, exact profile and prompt inspection, and a
|
||||
typed backend-capacity error;
|
||||
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
|
||||
adds application fallback profile filesystems and stops sending unset
|
||||
optional sampling controls as framework-selected provider values; and
|
||||
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||
defines the resulting profile-source and execution-setting precedence.
|
||||
|
||||
A source-compatibility test of the current Notarius repository against
|
||||
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
|
||||
intentional runtime and configuration migration rather than a repair for a
|
||||
breaking Go API change.
|
||||
|
||||
## Goals
|
||||
|
||||
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
|
||||
- Execute the exact prepared request snapshot whose safe details are recorded
|
||||
in Notarius debug material.
|
||||
- Validate configured PromptKit profiles through the upstream inspection API
|
||||
without synthetic prompts, provider calls, or credential-value access.
|
||||
- Give Notarius an application-owned, operator-overridable
|
||||
`dnd-extraction` profile fallback.
|
||||
- Let a pipeline choose one default LLM profile without repeating that ID on
|
||||
every LLM-backed binding.
|
||||
- Apply profile inheritance and run-wide overrides only where the resolved
|
||||
module or validator can use an LLM.
|
||||
- Preserve accurate checkpoint invalidation, effective profile provenance,
|
||||
redaction, cancellation, concurrency, and provider-neutral module contracts.
|
||||
- Provide operators with one clear deployment pattern for production,
|
||||
development, and local profile definitions.
|
||||
|
||||
## Target End State
|
||||
|
||||
### PromptKit Runtime Boundary
|
||||
|
||||
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
|
||||
reimplementing source or execution resolution.
|
||||
|
||||
For each structured completion, the adapter:
|
||||
|
||||
1. builds one PromptKit run request from the provider-neutral Notarius request;
|
||||
2. calls `PrepareExecution` once;
|
||||
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
|
||||
4. obtains credential-redacted `Details` for debug and response metadata; and
|
||||
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
|
||||
|
||||
The debug prompt and successful result therefore describe the same selected
|
||||
profile, rendered messages, input bytes, session, output contract, and effective
|
||||
settings even when a filesystem-backed source changes concurrently. PromptKit
|
||||
handle types remain private to `internal/framework/llm`.
|
||||
|
||||
PromptKit admission failures continue to match Notarius's provider-neutral
|
||||
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
|
||||
the adapter obtains the normalized backend ID through `errors.As` and may add it
|
||||
to safe application-owned diagnostics without parsing upstream error wording.
|
||||
The backend ID does not become a provider-specific module contract.
|
||||
|
||||
### Optional Provider Controls
|
||||
|
||||
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
|
||||
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
|
||||
requests and the provider chooses its own default. Notarius does not restore
|
||||
PromptKit's former implicit `top_p: 1` value globally.
|
||||
|
||||
An operator who requires a particular value specifies it in the selected
|
||||
PromptKit profile. The application fallback described below intentionally
|
||||
leaves these controls unset. A human-reviewed D&D extraction comparison should
|
||||
be performed after the upgrade, but paid or nondeterministic model output is
|
||||
not part of the default automated test suite.
|
||||
|
||||
### Profile Inspection
|
||||
|
||||
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
|
||||
effective explicit profile ID. It verifies that the profile exists, parses and
|
||||
validates, resolves its backend and target, and is compatible with the engine's
|
||||
registered backends. It does not create a synthetic prompt, load prompt inputs,
|
||||
contact a provider, or require credential values to exist in the validation
|
||||
process environment.
|
||||
|
||||
Credential availability is execution-time state. PromptKit preparation still
|
||||
enforces the selected profile's credential contract before generation. This
|
||||
keeps `notarius config validate` useful in build and deployment validation
|
||||
environments where secrets are deliberately absent.
|
||||
|
||||
PromptKit construction for inspection and execution uses one shared internal
|
||||
profile-source and backend-option path. The CLI does not expose PromptKit public
|
||||
types across the Notarius LLM boundary merely to perform inspection.
|
||||
|
||||
`InspectPrompt` is not adopted merely because it exists. It remains available
|
||||
for a later, separately defined module-to-prompt interface preflight if a
|
||||
concrete validation requirement justifies that additional contract.
|
||||
|
||||
### Application And Operator Profile Sources
|
||||
|
||||
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
|
||||
`dnd-extraction`. It is an application fallback registered through
|
||||
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
|
||||
preserves the current effective D&D baseline:
|
||||
|
||||
- backend: PromptKit's built-in `openrouter` backend;
|
||||
- model: `openai/gpt-5.6-luna`;
|
||||
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
|
||||
`medium`;
|
||||
- generation timeout: 240 seconds;
|
||||
- service tier: `flex`; and
|
||||
- no application-selected `temperature`, `max_tokens`, or `top_p`.
|
||||
|
||||
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
|
||||
`default_profile`. The ID communicates workload intent rather than a provider,
|
||||
model, or environment. Changing the embedded fallback is an intentional
|
||||
Notarius execution-policy change and participates in checkpoint identity.
|
||||
|
||||
Effective profile definitions resolve in PromptKit's order:
|
||||
|
||||
1. programmatic in-memory profiles used by tests or explicit consumers;
|
||||
2. the operator source configured by `promptkit.profile_file` or
|
||||
`promptkit.profile_dir`;
|
||||
3. the Notarius application fallback source; and
|
||||
4. PromptKit's embedded built-in catalog.
|
||||
|
||||
Only an absent ID falls through to the next source. A matching profile is a
|
||||
complete definition: fields are not merged with a lower-precedence definition,
|
||||
and a malformed matching operator profile fails rather than silently selecting
|
||||
the application fallback.
|
||||
|
||||
Production, development, and local deployments should normally provide
|
||||
different complete definitions for the same `dnd-extraction` ID. An operator
|
||||
source is optional because the application fallback keeps the maintained D&D
|
||||
workflow usable, but a deployment that needs an intentional model or backend
|
||||
policy should configure its own definition.
|
||||
|
||||
### Domain Ownership And Asset Assembly
|
||||
|
||||
The D&D fallback profile remains under `internal/modules/dnd` and is registered
|
||||
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
|
||||
to collect and flatten application fallback profile filesystems but contains no
|
||||
D&D model or policy knowledge.
|
||||
|
||||
The shared asset registry detects invalid roots, unreadable sources, and
|
||||
duplicate flattened paths. PromptKit remains responsible for strict profile
|
||||
YAML parsing, duplicate profile-ID detection, source precedence, and effective
|
||||
target resolution. The same assembled fallback source is supplied to runtime
|
||||
execution and CLI profile inspection.
|
||||
|
||||
### Explicit Module Execution Metadata
|
||||
|
||||
Every registered input, chunk, extract, merge, normalize, and output module
|
||||
declares one required execution class: `deterministic` or `llm_backed`.
|
||||
Validator registrations continue to declare the same distinction through their
|
||||
validator specifications.
|
||||
|
||||
The registered specification is authoritative for configuration resolution.
|
||||
Current production classifications are:
|
||||
|
||||
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
|
||||
LLM-backed;
|
||||
- the Seriatim input adapter, generic chunker, all current mergers, all other
|
||||
current normalizers, and the JSON output encoder are deterministic; and
|
||||
- current validators retain their declared classifications.
|
||||
|
||||
Missing or unsupported execution metadata is a registration error. Explicitly
|
||||
assigning `llm_profile` to a deterministic module or validator is a pipeline
|
||||
resolution error. The framework does not infer execution class by inspecting
|
||||
domain package names or concrete implementation types at runtime.
|
||||
|
||||
The module specification replaces the chunk runner's special runtime
|
||||
execution-class probe. Effective resolved bindings already express the result:
|
||||
only LLM-backed bindings may retain a non-empty profile.
|
||||
|
||||
### Pipeline-Level Profile Default
|
||||
|
||||
Configuration version 4 gains one optional non-empty pipeline field:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
dnd-session:
|
||||
llm_profile: dnd-extraction
|
||||
```
|
||||
|
||||
No configuration-version increment is required because the field is additive
|
||||
and existing files remain valid. An explicitly present blank value is invalid.
|
||||
|
||||
For every selected LLM-backed module and validator, the effective profile uses
|
||||
this precedence:
|
||||
|
||||
1. non-empty run-wide `--llm-profile` override;
|
||||
2. binding-specific `llm_profile`;
|
||||
3. pipeline-level `llm_profile`; and
|
||||
4. the prompt definition's `default_profile`, represented by an empty effective
|
||||
Notarius binding profile.
|
||||
|
||||
The run-wide override and inherited pipeline default never attach to a
|
||||
deterministic binding. Binding-specific exceptions remain available when one
|
||||
operation needs a different cost, latency, quality, backend, or reasoning
|
||||
policy.
|
||||
|
||||
Inheritance is resolved after module and validator selection, including
|
||||
`--only` lane filtering, but before effective-pipeline validation, digest
|
||||
construction, explicit-profile inspection, checkpoint construction,
|
||||
preparation, execution, or provenance capture. Only profiles used by selected
|
||||
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
|
||||
with no selected LLM-backed work does not require an otherwise unused profile
|
||||
to exist.
|
||||
|
||||
The resolved pipeline contains effective binding profiles rather than a second
|
||||
runtime inheritance mechanism. Two pipelines that differ only by spelling the
|
||||
same effective policy once as a pipeline default and once on every LLM-backed
|
||||
binding have the same semantic resolved digest. Changing an effective profile
|
||||
changes the digest and applicable checkpoint identity.
|
||||
|
||||
### Provenance And Checkpoints
|
||||
|
||||
The PromptKit profile-source checkpoint fingerprint covers:
|
||||
|
||||
- the PromptKit v0.5.0 built-in profile catalog identity;
|
||||
- exact application fallback profile asset content; and
|
||||
- exact configured operator profile YAML content, when present.
|
||||
|
||||
The existing local-backend target fingerprint remains separate and continues
|
||||
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
|
||||
stable markers, not profile contents, filesystem paths, endpoints, credentials,
|
||||
or other secrets.
|
||||
|
||||
Changing the PromptKit version, application fallback, operator profile, or
|
||||
effective pipeline profile makes incompatible LLM checkpoints ineligible for
|
||||
reuse. The dependency upgrade is expected to invalidate checkpoints produced
|
||||
under v0.3.0.
|
||||
|
||||
Successful run manifests continue to record only profiles actually selected by
|
||||
PromptKit, including their effective model, backend, and reasoning metadata.
|
||||
Debug output reports the same effective execution snapshot used for generation.
|
||||
|
||||
### Operator Documentation And Examples
|
||||
|
||||
Canonical documentation clearly distinguishes:
|
||||
|
||||
- Notarius prompt and schema assets embedded in the application;
|
||||
- Notarius application fallback profiles embedded in the application;
|
||||
- PromptKit's own embedded built-in profiles; and
|
||||
- operator profile files on the deployment filesystem.
|
||||
|
||||
The configuration reference owns the pipeline field, profile-source fields,
|
||||
validation rules, and precedence. Operations owns deployment layout, working
|
||||
directory behavior, credentials, and environment-specific profile management.
|
||||
The PromptKit integration document owns the pinned upstream contract and
|
||||
source-precedence boundary. Internal documents describe asset registration,
|
||||
resolution, inspection, prepared execution, fingerprinting, and tests without
|
||||
duplicating user-facing field definitions.
|
||||
|
||||
The maintained examples continue to include only the minimal and complete D&D
|
||||
configurations. They use the stable `dnd-extraction` policy, and one maintained
|
||||
PromptKit profile file under `examples/` demonstrates an operator override.
|
||||
Examples remain secret-free and are validated without live provider calls.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- Implementing the separate deterministic prompt-session identity roadmap
|
||||
item.
|
||||
- Changing the default `concurrency.total_llm` value; PromptKit's retained
|
||||
OpenRouter capacity of 16 remains relevant to that separate item.
|
||||
- Adding model evaluation as a deterministic or CI correctness gate.
|
||||
- Automatically selecting production, development, or local environments.
|
||||
Deployment configuration chooses the operator profile source.
|
||||
- Profile inheritance, partial profile merging, or cross-profile aliases.
|
||||
- Exposing PromptKit types to modules, validators, durable output contracts, or
|
||||
public configuration structures.
|
||||
- Adopting `InspectPrompt` without a separately justified prompt-interface
|
||||
validation contract.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
|
||||
- Every structured completion executes the exact snapshot used for safe debug
|
||||
prompt details.
|
||||
- Profile preflight uses profile inspection and no synthetic prompt.
|
||||
- The embedded `dnd-extraction` fallback resolves without an operator source,
|
||||
and a matching valid operator profile replaces it completely.
|
||||
- Every production module has explicit, correct execution metadata.
|
||||
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
|
||||
modules and validators, while deterministic bindings remain profile-free.
|
||||
- Effective profiles participate in pipeline digests, profile inspection,
|
||||
checkpoint identity, debug records, and run provenance at the appropriate
|
||||
boundaries.
|
||||
- The dependency and application fallback changes invalidate incompatible old
|
||||
checkpoints without exposing profile or credential content.
|
||||
- Canonical documentation and maintained examples accurately describe and
|
||||
exercise the implemented operator workflow.
|
||||
- Default tests remain deterministic, offline, credential-free, and focused on
|
||||
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.
|
||||
@@ -249,10 +249,24 @@ func TestRunAutoReusesPlanWhenRunInputsChange(t *testing.T) {
|
||||
}
|
||||
harness.mu.Lock()
|
||||
chunkCalls := harness.chunkCalls
|
||||
sessions := append([]string(nil), harness.sessionIDs...)
|
||||
harness.mu.Unlock()
|
||||
if chunkCalls != 1 {
|
||||
t.Fatalf("chunk calls across changed run inputs = %d, want 1", chunkCalls)
|
||||
}
|
||||
rawInput, err := os.ReadFile(roots.input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wantSessionID, err := resolvePromptSessionID("", "test/input", rawInput)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, sessionID := range sessions {
|
||||
if sessionID != wantSessionID {
|
||||
t.Fatalf("session IDs across reference changes = %#v, want %q", sessions, wantSessionID)
|
||||
}
|
||||
}
|
||||
assertFile(t, filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"), "plan.json"))
|
||||
assertAnyFile(t, roots.output)
|
||||
}
|
||||
|
||||
@@ -24,6 +24,9 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
||||
for _, example := range maintainedExampleFiles(t) {
|
||||
t.Run(example.name, func(t *testing.T) {
|
||||
cfg := loadMaintainedExample(t, example.path)
|
||||
if example.name == "complete" && (cfg.Concurrency.TotalLLM != 2 || cfg.Concurrency.StageWorkers["extract"] != 2) {
|
||||
t.Fatalf("complete example concurrency = %#v, want explicit limits of 2", cfg.Concurrency)
|
||||
}
|
||||
raw, err := os.ReadFile(example.transcriptPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read maintained transcript %q: %v", example.transcriptPath, err)
|
||||
@@ -242,6 +245,12 @@ func TestMaintainedMalformedInputOnlyRecordsDebugFailureWhenRequested(t *testing
|
||||
if report.Succeeded || report.PipelineID != "dnd-session" {
|
||||
t.Fatalf("failure report = %#v, want failed dnd-session report", report)
|
||||
}
|
||||
invocation := readProductionJSON[debugbundle.Invocation](t, filepath.Join(bundle, "summary", "invocation.json"))
|
||||
manifest := readProductionJSON[artifacts.RunManifest](t, filepath.Join(bundle, "summary", "run-manifest.json"))
|
||||
manifestSession, found := manifest.Metadata["session_id"]
|
||||
if invocation.SessionID == "" || !found || manifestSession != invocation.SessionID {
|
||||
t.Fatalf("failed run sessions: invocation=%q manifest=%#v metadata=%#v", invocation.SessionID, manifestSession, manifest.Metadata)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -153,10 +153,10 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
|
||||
recomputeStep := singleValueFlag{name: "--recompute-step"}
|
||||
chunkCache := chunkCacheFlag{}
|
||||
sessionID := sessionIDFlag{}
|
||||
requestedSessionID := sessionIDFlag{}
|
||||
referenceFlags := stringListFlag{}
|
||||
withoutReferenceFlags := stringListFlag{}
|
||||
fs.Var(&sessionID, "session-id", "prompt session identifier")
|
||||
fs.Var(&requestedSessionID, "session-id", "prompt session identifier")
|
||||
fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override")
|
||||
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
|
||||
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
|
||||
@@ -199,7 +199,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
fmt.Fprintln(stderr, "notarius: --debug-dir must not be empty")
|
||||
return 2
|
||||
}
|
||||
if sessionID.set && strings.TrimSpace(sessionID.value) == "" {
|
||||
if requestedSessionID.set && strings.TrimSpace(requestedSessionID.value) == "" {
|
||||
fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
|
||||
return 2
|
||||
}
|
||||
@@ -414,11 +414,19 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
|
||||
}
|
||||
effectiveSessionID, err := resolvePromptSessionID(requestedSessionID.value, effective.ResolvedPipeline.Input.Module, rawInput)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
invocation.SessionID = effectiveSessionID
|
||||
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
|
||||
}
|
||||
chunkPlans, err := chunkPlanStoreForRun(effective.Config.Cache.ChunkPlans, opts)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), runtimeOverrides, *resume)
|
||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), effectiveSessionID, runtimeOverrides, *resume)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
@@ -427,7 +435,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
Prepared: prepared,
|
||||
Path: strings.TrimSpace(*inputPath),
|
||||
RawInput: rawInput,
|
||||
SessionID: strings.TrimSpace(sessionID.value),
|
||||
SessionID: effectiveSessionID,
|
||||
RunID: runID,
|
||||
StartedAt: startedAt,
|
||||
LLMProfiles: llmProfiles,
|
||||
|
||||
@@ -492,19 +492,30 @@ func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunSessionIDUsesExplicitValueOrSourceDocumentID(t *testing.T) {
|
||||
func TestRunSessionIDUsesEffectiveValueForPromptRequests(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{name: "source document", want: "source"},
|
||||
{name: "derived default"},
|
||||
{name: "explicit trimmed value", args: []string{"--session-id", " explicit-session "}, want: "explicit-session"},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
want := tt.want
|
||||
if want == "" {
|
||||
rawInput, err := os.ReadFile(roots.input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want, err = resolvePromptSessionID("", "test/input", rawInput)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
harness := newStateTestHarness()
|
||||
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.args...)
|
||||
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--debug"}, tt.args...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := RunWithOptions(args, &stdout, &stderr, harness.options())
|
||||
if code != 0 || stderr.Len() != 0 {
|
||||
@@ -517,10 +528,15 @@ func TestRunSessionIDUsesExplicitValueOrSourceDocumentID(t *testing.T) {
|
||||
t.Fatalf("session IDs = %#v, want all prompt-facing module requests", sessions)
|
||||
}
|
||||
for _, session := range sessions {
|
||||
if session != tt.want {
|
||||
t.Fatalf("session IDs = %#v, want %q", sessions, tt.want)
|
||||
if session != want {
|
||||
t.Fatalf("session IDs = %#v, want %q", sessions, want)
|
||||
}
|
||||
}
|
||||
var manifest artifacts.RunManifest
|
||||
readStateTestSummaryJSON(t, onlyChildDir(t, roots.debug), "run-manifest.json", &manifest)
|
||||
if session, ok := manifest.Metadata["session_id"]; !ok || session != want {
|
||||
t.Fatalf("manifest session = %#v, want %q; metadata = %#v", session, want, manifest.Metadata)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
26
internal/cli/session.go
Normal file
26
internal/cli/session.go
Normal file
@@ -0,0 +1,26 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const generatedSessionIDPrefix = "notarius:v1:"
|
||||
|
||||
func resolvePromptSessionID(explicitSessionID, inputModule string, rawInput []byte) (string, error) {
|
||||
inputModule = strings.TrimSpace(inputModule)
|
||||
if inputModule == "" {
|
||||
return "", fmt.Errorf("resolve prompt session: input module key must not be empty")
|
||||
}
|
||||
if sessionID := strings.TrimSpace(explicitSessionID); sessionID != "" {
|
||||
return sessionID, nil
|
||||
}
|
||||
|
||||
hasher := sha256.New()
|
||||
_, _ = hasher.Write([]byte(inputModule))
|
||||
_, _ = hasher.Write([]byte{0})
|
||||
_, _ = hasher.Write(rawInput)
|
||||
return generatedSessionIDPrefix + hex.EncodeToString(hasher.Sum(nil)), nil
|
||||
}
|
||||
61
internal/cli/session_test.go
Normal file
61
internal/cli/session_test.go
Normal file
@@ -0,0 +1,61 @@
|
||||
package cli
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestResolvePromptSessionIDUsesVersionedInputIdentity(t *testing.T) {
|
||||
got, err := resolvePromptSessionID("", " seriatim/input/transcript ", []byte("{\"entries\":[\"one\"]}\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const want = "notarius:v1:e15fefdca48653e73248b4157900547be1fd34250c0138f8d3d1f2e89b43bb25"
|
||||
if got != want {
|
||||
t.Fatalf("resolved session = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePromptSessionIDStabilityAndOverride(t *testing.T) {
|
||||
rawInput := []byte("same input")
|
||||
baseline, err := resolvePromptSessionID("", "input/transcript", rawInput)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
repeated, err := resolvePromptSessionID("", "input/transcript", rawInput)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if baseline != repeated {
|
||||
t.Fatalf("resolved sessions = %q and %q, want stable value", baseline, repeated)
|
||||
}
|
||||
|
||||
differentModule, err := resolvePromptSessionID("", "input/other", rawInput)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if baseline == differentModule {
|
||||
t.Fatalf("resolved sessions = %q for distinct input modules", baseline)
|
||||
}
|
||||
|
||||
differentInput, err := resolvePromptSessionID("", "input/transcript", []byte("same inpuu"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if baseline == differentInput {
|
||||
t.Fatalf("resolved sessions = %q for distinct input bytes", baseline)
|
||||
}
|
||||
|
||||
override, err := resolvePromptSessionID(" explicit-session ", "input/other", []byte("different input"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if override != "explicit-session" {
|
||||
t.Fatalf("resolved override = %q, want %q", override, "explicit-session")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePromptSessionIDRejectsEmptyInputModule(t *testing.T) {
|
||||
for _, explicitSessionID := range []string{"", "explicit-session"} {
|
||||
if _, err := resolvePromptSessionID(explicitSessionID, " \t", []byte("input")); err == nil {
|
||||
t.Fatalf("resolvePromptSessionID(%q) error = nil, want empty module failure", explicitSessionID)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -329,11 +329,25 @@ func TestRunUsesOneInjectedIdentityForDebugOutputAndManifest(t *testing.T) {
|
||||
if manifest.StartedAt == nil || !manifest.StartedAt.Equal(wantStartedAt) {
|
||||
t.Fatalf("manifest started at = %v, want %v", manifest.StartedAt, wantStartedAt)
|
||||
}
|
||||
rawInput, err := os.ReadFile(roots.input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wantSessionID, err := resolvePromptSessionID("", "test/input", rawInput)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sessionID, ok := manifest.Metadata["session_id"]; !ok || sessionID != wantSessionID {
|
||||
t.Fatalf("manifest session = %#v, want %q; metadata = %#v", sessionID, wantSessionID, manifest.Metadata)
|
||||
}
|
||||
var invocation debugbundle.Invocation
|
||||
readStateTestSummaryJSON(t, debugPath, "invocation.json", &invocation)
|
||||
if invocation.RunID != runID || !invocation.StartedAt.Equal(wantStartedAt) {
|
||||
t.Fatalf("debug invocation identity = %#v, want run %q at %v", invocation, runID, wantStartedAt)
|
||||
}
|
||||
if invocation.SessionID != wantSessionID {
|
||||
t.Fatalf("debug invocation session = %q, want %q", invocation.SessionID, wantSessionID)
|
||||
}
|
||||
report := readStateTestRunReport(t, debugPath)
|
||||
if !report.Succeeded || report.RunID != runID || report.PipelineID != "sample" || report.OutputPath != outputPath || report.DebugPath != debugPath || report.OutputCount != 1 || report.RejectedCount != 0 || report.WarningCount != 0 || report.ValidationStatus != "approved" {
|
||||
t.Fatalf("success report = %#v", report)
|
||||
@@ -343,6 +357,45 @@ func TestRunUsesOneInjectedIdentityForDebugOutputAndManifest(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunCheckpointReuseRequiresSameEffectiveSession(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
harness := newStateTestHarness()
|
||||
run := func(extra ...string) stateTestResult {
|
||||
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
|
||||
args = append(args, extra...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
return stateTestResult{code: RunWithOptions(args, &stdout, &stderr, harness.options()), stdout: stdout.String(), stderr: stderr.String()}
|
||||
}
|
||||
|
||||
if result := run(); result.code != 0 {
|
||||
t.Fatalf("initial run code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
|
||||
}
|
||||
harness.mu.Lock()
|
||||
initialExtractCalls := harness.extractCalls
|
||||
harness.mu.Unlock()
|
||||
if initialExtractCalls != 1 {
|
||||
t.Fatalf("initial extract calls = %d, want 1", initialExtractCalls)
|
||||
}
|
||||
if result := run("--resume"); result.code != 0 {
|
||||
t.Fatalf("same-session resume code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
|
||||
}
|
||||
harness.mu.Lock()
|
||||
reusedExtractCalls := harness.extractCalls
|
||||
harness.mu.Unlock()
|
||||
if reusedExtractCalls != initialExtractCalls {
|
||||
t.Fatalf("extract calls after same-session resume = %d, want %d", reusedExtractCalls, initialExtractCalls)
|
||||
}
|
||||
if result := run("--resume", "--session-id", "different-session"); result.code != 0 {
|
||||
t.Fatalf("different-session resume code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
|
||||
}
|
||||
harness.mu.Lock()
|
||||
differentSessionExtractCalls := harness.extractCalls
|
||||
harness.mu.Unlock()
|
||||
if differentSessionExtractCalls != initialExtractCalls+1 {
|
||||
t.Fatalf("extract calls after different-session resume = %d, want %d", differentSessionExtractCalls, initialExtractCalls+1)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunWritesTerminalArtifactsForResolutionPipelineAndOutputFailures(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
|
||||
@@ -34,6 +34,8 @@ type ConcurrencyConfig struct {
|
||||
defaultedExtractWorkers int
|
||||
}
|
||||
|
||||
const defaultLLMConcurrency = 16
|
||||
|
||||
type OutputConfig struct {
|
||||
Directory string `json:"directory"`
|
||||
}
|
||||
@@ -60,9 +62,9 @@ func Default() Config {
|
||||
return Config{
|
||||
Pipelines: map[string]pipeline.PipelineProfile{},
|
||||
Concurrency: ConcurrencyConfig{
|
||||
TotalLLM: 1,
|
||||
StageWorkers: map[string]int{"extract": 1},
|
||||
defaultedExtractWorkers: 1,
|
||||
TotalLLM: defaultLLMConcurrency,
|
||||
StageWorkers: map[string]int{"extract": defaultLLMConcurrency},
|
||||
defaultedExtractWorkers: defaultLLMConcurrency,
|
||||
},
|
||||
Output: OutputConfig{Directory: "./notarius-output"},
|
||||
Cache: CacheConfig{ChunkPlans: ChunkPlanCacheConfig{Mode: pipeline.ChunkCacheAuto}},
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
|
||||
func TestDefaultReturnsDocumentedValuesAndIndependentMaps(t *testing.T) {
|
||||
first := Default()
|
||||
if first.Concurrency.TotalLLM != 1 || first.Concurrency.StageWorkers["extract"] != 1 {
|
||||
if first.Concurrency.TotalLLM != 16 || first.Concurrency.StageWorkers["extract"] != 16 {
|
||||
t.Fatalf("concurrency defaults = %#v", first.Concurrency)
|
||||
}
|
||||
if first.Output.Directory != "./notarius-output" || first.Debug.Directory != "./notarius-debug" {
|
||||
@@ -31,7 +31,7 @@ func TestDefaultReturnsDocumentedValuesAndIndependentMaps(t *testing.T) {
|
||||
first.Concurrency.StageWorkers["other"] = 100
|
||||
first.Pipelines["changed"] = pipeline.PipelineProfile{}
|
||||
second := Default()
|
||||
if second.Concurrency.StageWorkers["extract"] != 1 || len(second.Concurrency.StageWorkers) != 1 || len(second.Pipelines) != 0 {
|
||||
if second.Concurrency.StageWorkers["extract"] != 16 || len(second.Concurrency.StageWorkers) != 1 || len(second.Pipelines) != 0 {
|
||||
t.Fatalf("Default() returned state shared with an earlier result: %#v", second)
|
||||
}
|
||||
}
|
||||
@@ -45,7 +45,7 @@ func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
|
||||
if cfg.Output.Directory != "./notarius-output" || cfg.Debug.Directory != "./notarius-debug" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheAuto {
|
||||
t.Fatalf("minimal file changed unrelated defaults: %#v", cfg)
|
||||
}
|
||||
if cfg.Concurrency.TotalLLM != 1 || cfg.Concurrency.StageWorkers["extract"] != 1 || len(cfg.Pipelines) != 0 {
|
||||
if cfg.Concurrency.TotalLLM != 16 || cfg.Concurrency.StageWorkers["extract"] != 16 || len(cfg.Pipelines) != 0 {
|
||||
t.Fatalf("minimal file did not retain defaults: %#v", cfg)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -179,6 +179,30 @@ func TestWriteInvocationPreservesReasoningEffortOverrideStates(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteInvocationOmitsEmptySessionID(t *testing.T) {
|
||||
for _, sessionID := range []string{"", "session-123"} {
|
||||
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := bundle.Summary().WriteInvocation(Invocation{Operation: "run", SessionID: sessionID}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(bundle.SummaryRoot(), ArtifactInvocationMetadata))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value, found := payload["session_id"]
|
||||
if found != (sessionID != "") || (found && value != sessionID) {
|
||||
t.Fatalf("session found=%t value=%#v, want found=%t value=%q; JSON=%s", found, value, sessionID != "", sessionID, data)
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
|
||||
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||
if err != nil {
|
||||
|
||||
@@ -40,6 +40,7 @@ type Invocation struct {
|
||||
OnlyLanes []string `json:"only_lanes,omitempty"`
|
||||
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
||||
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
RunID string `json:"run_id"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
}
|
||||
|
||||
@@ -91,6 +91,11 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
input.llmClient = input.Prepared.dependencies.LLM
|
||||
|
||||
output.Manifest = manifestFromPipeline(input)
|
||||
sessionID := strings.TrimSpace(input.SessionID)
|
||||
output.Manifest.Metadata, err = manifestMetadataWithSessionID(output.Manifest.Metadata, sessionID)
|
||||
if err != nil {
|
||||
return failOutput(output), err
|
||||
}
|
||||
output.ChunkPlan = &artifacts.ChunkPlanSummary{
|
||||
Mode: string(effectiveChunkCacheMode(input.ChunkCacheMode)), RequestedModule: input.pipeline.Chunk.Module,
|
||||
LookupStatus: "skipped", LookupReason: "chunk plan lookup skipped",
|
||||
@@ -191,11 +196,6 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
return failOutput(output), fmt.Errorf("write source debug artifact: %w", err)
|
||||
}
|
||||
sourceInput := sourceInputMaterial(input.Path, input.RawInput)
|
||||
sessionID := resolvedSessionID(input.SessionID, doc.ID)
|
||||
output.Manifest.Metadata, err = manifestMetadataWithSessionID(output.Manifest.Metadata, sessionID)
|
||||
if err != nil {
|
||||
return failOutput(output), err
|
||||
}
|
||||
output.Manifest.SourceDigests = []string{doc.Digest}
|
||||
|
||||
chunker := input.Prepared.chunker
|
||||
@@ -896,13 +896,6 @@ func sourceInputOriginURI(inputPath string) string {
|
||||
return fileURI(inputPath)
|
||||
}
|
||||
|
||||
func resolvedSessionID(explicit string, sourceDocumentID string) string {
|
||||
if trimmed := strings.TrimSpace(explicit); trimmed != "" {
|
||||
return trimmed
|
||||
}
|
||||
return strings.TrimSpace(sourceDocumentID)
|
||||
}
|
||||
|
||||
func manifestMetadataWithSessionID(metadata map[string]any, sessionID string) (map[string]any, error) {
|
||||
out, err := cloneMetadata(metadata)
|
||||
if err != nil {
|
||||
|
||||
37
internal/framework/pipeline/runner_session_test.go
Normal file
37
internal/framework/pipeline/runner_session_test.go
Normal file
@@ -0,0 +1,37 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRunnerUsesProvidedSessionWithoutSourceFallback(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
session string
|
||||
wantSet bool
|
||||
want string
|
||||
}{
|
||||
{name: "provided", session: " supplied-session ", wantSet: true, want: "supplied-session"},
|
||||
{name: "empty", wantSet: false},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
prepared, _ := preparedTerminalDebugPipeline(t)
|
||||
output, err := New().Run(context.Background(), RunInput{
|
||||
Prepared: prepared,
|
||||
RawInput: []byte("input"),
|
||||
SessionID: test.session,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value, found := output.Manifest.Metadata["session_id"]
|
||||
if found != test.wantSet {
|
||||
t.Fatalf("manifest session presence = %t, want %t; metadata = %#v", found, test.wantSet, output.Manifest.Metadata)
|
||||
}
|
||||
if found && value != test.want {
|
||||
t.Fatalf("manifest session = %#v, want %q", value, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user