diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..56b66f3 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,220 @@ +# Message-Content Hash Implementation Plan + +## Objective And Authority + +Implement the target in [message-content-hash.md](message-content-hash.md). +That roadmap owns scope, hash semantics, and the accepted cross-release +compatibility decision. The [Notarius request](notarius-request.md) is background; +its application workflow does not add Promptkit requirements. + +This is one end-to-end sprint, divided into four sequential, single-prompt +stages sized for a Luna coding agent. Intermediate stages need not compile or +leave a releasable repository. There are no intermediate human-review or +external-publication gates. The final stage verifies the assembled result. +Do not introduce temporary public APIs or compatibility shims just to make an +intermediate stage compile. + +Start with [development.md](../development.md) and its applicable policies: +[agents](../policy/agents.md), [architecture](../policy/architecture.md), +[testing](../policy/testing.md), and [documentation](../policy/documentation.md). +Use the [release policy](../policy/release.md) when preparing release notes. +Follow repository commit policy for completed units; commits are not approval +gates between stages. + +## Baseline And Fixed Design Choices + +The reviewed baseline is Promptkit v0.23.3 plus the accepted roadmap commits. + +- `internal/usecase/runner.go` assembles the final message sequence in + `completePreparationWithStructuredOutput`, then builds `domain.PreparedRun`. + `hashRenderedPrompt` includes session and cache metadata and stays unchanged. +- `internal/usecase/prepared_execution.go` shares the prepared snapshot between + retained details and execution payload. Public `Details` detaches through + `fromDomainPreparedRun` in `convert.go`; `OnPrepared` uses that conversion too. +- `PreparedRun` in `types.go` is a public data value with stable JSON field names. + Its JSON marshaler in `json.go` uses the defined type + `preparedRunJSONFields PreparedRun`, not a separately maintained field list. + +Select these concrete implementation details for this sprint: + +1. Add `MessageContentHash string` to internal and public `PreparedRun`. + The public tag is `json:"message_content_hash"`, without `omitempty`. + Zero/manual values retain the ordinary empty string; preparation populates + the field. JSON decoding stores the supplied field rather than recomputing + it or validating caller-owned message/hash consistency. Older JSON that + omits the field decodes with the ordinary zero value. +2. Compute the value once from the final prepared messages using the v1 scheme + below. Carry the string through existing snapshot and conversion paths. + Reading details does not rehash caller-modified copies or claim a handle. +3. Keep the existing rendered hash and all its values unchanged. Do not add the + new field to `RunResult`, `Exchange`, model requests, HTTP payloads, or + metadata observer events merely for symmetry. Existing `OnPrepared` values + include it because they are `PreparedRun` values. +4. Add no hashing option, standalone public hash function, dependency, timer, + storage feature, or application-specific interpretation. Preparation and + execution retain their existing validation, ownership, and lifecycle rules. + +## Stage 1 — Implement The Versioned Hash And Reference Tests + +Work in `internal/usecase`, using a focused `message_content_hash.go` and +`message_content_hash_test.go`. Implement the unexported helper +`hashMessageContent(messages []domain.RenderedMessage) string`. + +The exact v1 preimage is the concatenation of: + +1. The literal ASCII prefix `promptkit/message-content/v1` followed by one NUL + byte, expressed in Go as `"promptkit/message-content/v1\x00"`. +2. The message count as an unsigned 64-bit big-endian integer. +3. For each message in order: the role byte length as an unsigned 64-bit + big-endian integer, the role bytes, then the content byte length in the same + format and the content bytes. String lengths are Go `len(string)` byte + counts, not Unicode code-point counts. + +Hash that preimage with SHA-256 and return lowercase hexadecimal. This is an +internal scheme specification, not a public format callers should parse. +Include no session, cache-control, or other fields. Do not normalize strings +or add validation to this helper; preparation already owns input validity. + +Stream the prefix and fields into the hasher. Reuse the existing +`writeRenderedPromptHashLength` and `writeRenderedPromptHashString` helpers +without renaming or changing their behavior. A second linear hash traversal +is sufficient; do not refactor the existing hash into a generic policy engine +or build an aggregate serialized message buffer. + +Protect the v1 scheme with these independently calculated fixed vectors: + +| Messages, with strings in Go-style escaped notation | Expected hash | +| --- | --- | +| Empty slice | `4b3d27f1ccb333598f041ab4c079f285ba65b095066dda9064e23e0de57116c7` | +| `[("user", "hello")]` | `34e867fb7cfe57c68d3a25ce6f007751230b3c41915d927492d96158c6cb0f04` | +| `[("system", "a\x00b"), ("user", "café\n")]` | `38bcd39c878e656be34449951e24db771a5fb14d809d54b918b0c18b79c1925f` | + +Use literal expected digests, not expectations generated by the production +helper. The empty-vector case tests framing only; it does not authorize empty +prompt definitions. Add a compact relational table covering role, text, +whitespace, order, count, and boundary changes, plus cache-control invariance. +Include a one-message/two-message ambiguous-concatenation pair. Nil and empty +slices both encode a zero count. Do not add subprocess tests just to prove a +deterministic pure function; fixed vectors protect cross-process stability. + +Validation: run the new helper tests and the existing `TestHashRenderedPrompt` +tests in `internal/usecase`. Completion means the v1 vectors and semantic +relations pass without changing existing rendered-hash expectations. + +## Stage 2 — Wire Prepared Values And Snapshot Lifecycle + +Update `internal/domain/domain.go`, `types.go`, `internal/usecase/runner.go`, +and `convert.go`: + +- Add the internal/public field beside `RenderedPromptHash` and document its + public semantics in GoDoc. Cover role/text inputs, excluded metadata, + opaque filename-safe output, preparation-time computation, and the accepted + cross-release compatibility rule. Do not promise a permanently fixed scheme. +- Populate it in `completePreparationWithStructuredOutput` from + `effectivePrompt.Messages`, after history, rendered messages, and appended + messages have been assembled. Do not hash raw definitions or an intermediate + suffix, and do not add another preparation or source-loading pass. +- Copy it in `fromDomainPreparedRun`. Check the existing snapshot path rather + than creating another metadata snapshot, getter computation, or locking layer. + `Prepare`, `Details`, ordinary/prepared `OnPrepared`, and ordinary execution's + shared preparation should all use this captured value. + +Update `json_contract_test.go`: populate the field in +`fullyPopulatedPreparedRun`, assert the `message_content_hash` member in its +round-trip test, and check empty-string encoding on a zero value and zero-value +decoding when the member is absent. The defined-type JSON projection should +inherit the field; do not create a new serialization DTO just for this change. + +Extend the relevant tests in `prepared_execution_contract_test.go` and +`prepared_notification_contract_test.go`, or add a small focused external-package +test file. Verify: + +- Successful `Prepare` and `PrepareExecution(...).Details()` agree for identical + effective messages and expose a nonempty value before any model call. +- Mutating the returned details' messages and hash does not alter later details + or execution. Reading the field does not consume a handle, and retained details + still expose it after execution or discard under the existing lifecycle. +- `OnPrepared` receives the same captured hash on ordinary and split execution, + with no additional preparation/notification call introduced for hashing. + +Reuse existing recording clients and ownership fixtures. Do not duplicate their +complete lifecycle, concurrency, credential, or repair test inventories. + +Validation: run the focused prepared-value, notification, and JSON contract +tests. Completion means the value survives all supported prepared projections +and existing handle ownership remains intact. + +## Stage 3 — Verify Public Content Semantics And Exclusions + +Add focused `promptkit_test` coverage, preferably in +`message_content_hash_contract_test.go`, using real engines with small synthetic +definitions and an injected model client only where history or execution is +needed. Keep cases grouped by distinct risks rather than building a Cartesian +product of every setting and API. + +1. **Equality and exclusions:** prepare the same role/text sequence repeatedly + and in separate engines. Vary session ID and cache hints and verify the new + hash stays equal while the existing rendered hash changes. Independently + vary prompt/profile/backend identifiers, effective model/settings, direct + synthetic credentials, and output/schema metadata outside message text; + verify unchanged prepared role/text yields the same new hash. Use a valid + in-memory schema where necessary. Avoid sleeps merely to vary timing. +2. **Rendered contributions:** use a template with a variable and input text. + Changes to rendered bytes change the hash; changes to unused variables or + artifact URI/hash metadata alone do not when rendered messages are unchanged. + Reuse existing input helper syntax and in-memory artifact fixtures. +3. **Composition:** exercise appended messages and a captured exchange. Compare + against a separately prepared explicit sequence with identical roles/text + and no history identity; hashes agree. Changing a history/appended role or + text changes the value. This proves the complete effective sequence is hashed, + not just the newly rendered definition. Initial history capture may execute + a fake client; preparation itself must not execute it. + +Assert semantic equality/difference rather than duplicating the hash algorithm +in facade tests. Stage 1 owns exact v1 bytes; facade tests own message selection +and preparation integration. Retain existing rendered-hash tests unchanged. +Do not test or implement any application use of the hash. + +Validation: run the new contract tests alongside existing appended-message and +exchange/preparation tests. Completion means every included/excluded category +in the feature roadmap has proportionate coverage at its owning boundary. + +## Stage 4 — Document, Demonstrate, And Validate The End State + +Update these existing documentation owners: + +- `docs/consumers/pkg-promptkit.md`: explain the narrower hash versus + `RenderedPromptHash`, and show reading `details.MessageContentHash` after + `PrepareExecution` with ordinary error handling and deferred `Discard`. + Keep the example neutral about application interpretation. +- `docs/internal/runner.md`: describe the preparation point, content-only + preimage, streaming computation, and v1 scheme stability/reference tests. + Public GoDoc remains the exact API contract owner. +- `examples/go-library/prepare/main.go` and its test: include the new hash in + the existing preparation summary and assert it is nonempty. Do not hard-code + an example digest or introduce an application workflow or another example + project. Existing offline preparation behavior remains unchanged. +- Prepare `docs/releases/v0.24.0.md` for this new feature and link it from the + README. Describe the new prepared-result JSON member, no configuration + migration, and the additive Go struct field (positional literals may require + adjustment). Use the normal licensing footer. Do not edit published notes, + tag, push, or deliver a release as part of this implementation sprint. + +Use the newest installed Go toolchain following `docs/development.md`, with +`GOTOOLCHAIN=local` and `GOWORK=off`. Keep dependency versions and the consumer +minimum unchanged for this feature. Format changed Go files, run +`go test -race ./...`, `go vet ./...`, and `git diff --check`. The complete Go +suite includes maintained example tests; do not add a separate repeated example +gate. If `/tmp` is still full, select an isolated writable `TMPDIR` and build +cache on a filesystem with space rather than deleting unrelated files. + +Review the final diff against the feature roadmap: both hashes have their +distinct semantics, all prepared projections carry the new value, no new model +request field or application behavior has appeared, and reference vectors retain +the selected scheme. Check new local documentation links. Use focused reruns +while fixing failures and finish with validation appropriate to the final code. +No upstream dependency publication or consumer-project changes are needed. + +Completion is the implemented and documented hash, successful final validation, +and reviewed commits under repository policy. Retire the completed planning +documents during normal release preparation, not as an extra execution gate. diff --git a/docs/roadmap/message-content-hash.md b/docs/roadmap/message-content-hash.md index 272bf32..79f59f7 100644 --- a/docs/roadmap/message-content-hash.md +++ b/docs/roadmap/message-content-hash.md @@ -11,7 +11,7 @@ semantics. Applications own its interpretation and use. Downstream caching is one motivation, not a Promptkit feature or a requirement on application behavior. This roadmap defines the target capability and the work needed from the current -baseline. It is not a staged implementation plan. +baseline. The [implementation plan](implementation.md) owns staged execution. ## Current Baseline diff --git a/docs/roadmap/notarius-request.md b/docs/roadmap/notarius-request.md new file mode 100644 index 0000000..0707141 --- /dev/null +++ b/docs/roadmap/notarius-request.md @@ -0,0 +1,143 @@ +# Notarius Request: Message-Content Hashing + +**Status:** Requested by the Notarius consumer; proposed capability, not an +implemented API or an approved PromptKit implementation plan. + +**Reviewed baseline:** PromptKit v0.23.3. + +## Requested Capability + +Expose a deterministic hash of a prepared request's ordered message roles and +exact text, available before model execution. `MessageContentHash` is the +suggested name. This would let consumers reuse an accepted answer to the same +message content without implementing their own message-hashing policy. + +A field on `PreparedRun`, available through ordinary preparation and +`PreparedExecution.Details()`, appears sufficient. The exact public API is for +PromptKit to select; Notarius needs the value at the preparation boundary, not +only after execution or through an execution observer. + +## Consumer Need + +Notarius runs pipelines whose stages depend on artifacts from earlier stages. +If a later stage fails, an operator should be able to fix the problem and rerun +without paying for successful earlier model work again. Notarius is replacing +its checkpoint machinery with a small accepted-artifact cache. + +The cached value is **an accepted answer to the original task**. Notarius will +prepare the initial request, obtain its message-content hash, and look for +`cache/.json`. A hit loads the accepted domain artifact directly. A miss +executes the full application operation, including domain interpretation, +validation, and any application-directed correction passes. Only the final +accepted artifact is stored, under the original request's hash. + +This is intentionally a convenient reuse policy, not proof that every execution +setting, validation policy, or application implementation is identical. A change +to model settings alone should not invalidate an otherwise reusable answer. +Operators can bypass or refresh the application cache when they want fresh work. + +## Hash Semantics + +Hash the complete effective message sequence exposed by native preparation: +rendered definition messages, appended messages, and captured conversation +history when present. Include each message's role and exact content, preserving +message order and boundaries. + +| Included | Excluded | +| --- | --- | +| Ordered message roles and exact text | Session, run, and parent-run identity; timestamps and timings | +| Message boundaries and sequence length | Provider cache-control hints and other non-content message metadata | +| Any input, instruction, or schema text actually rendered into messages | Model, backend, profile, credentials, generation settings, timeouts, budgets, and retry/repair settings | +| | Prompt ID, definition hash, input hashes/paths, and schema/output-contract metadata outside message text | + +No whitespace trimming, Unicode normalization, or semantic equivalence is +requested: hash the exact prepared text. Different roles, ordering, boundaries, +or text should produce different hashes, subject to ordinary hash collision +properties. Identical role/text sequences should produce the same value across +requests, engines, and process invocations regardless of excluded metadata. +For example, changing a prompt ID while retaining identical messages should +preserve the hash. Changing settings that also changes rendered text should +change it because the text changed. + +Return an opaque, filename-safe value suitable for a persisted cache key. +PromptKit should own and document the hashing semantics and compatibility +behavior. A deterministic, unambiguous encoding and a standard hash such as +SHA-256 would suffice; the precise encoding is an upstream implementation choice. +No general fingerprint configuration or selectable hashing-policy framework is +needed for this request. + +## Existing APIs And The Gap + +[`PreparedRun`](../../types.go) already exposes messages and +`RenderedPromptHash`. The latter includes session identity and message cache +hints, as described in the [runner documentation](../internal/runner.md). +Hashing the entire prepared value would additionally capture settings and +volatile preparation metadata. Neither supplies the requested content-only key. + +Keep the existing hash's semantics intact and add the narrower capability. +Notarius could manually project and hash message fields, but that would preserve +the duplicated library mechanics this migration aims to remove. + +[Native split preparation and execution](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later) +already provides the needed execution boundary. Preparation makes no model call +and reserves no backend capacity. On a miss, `RunPrepared` can execute the retained +snapshot. On a hit, the application can discard the handle. No new execution +protocol is requested. + +## Intended Application Flow + +The following is conceptual pseudocode, not an existing hash API: + +```text +prepared = engine.PrepareExecution(initialRequest) +initialKey = prepared.Details().MessageContentHash // proposed addition + +if cache contains a usable accepted artifact for initialKey: + discard prepared handle + return cached artifact + +result = engine.RunPrepared(prepared) +artifact = interpret, validate, and correct through the Notarius operation +if the operation produces a final accepted artifact: + save artifact to cache under initialKey +return application outcome +``` + +Notarius retains the initial key even when acceptance requires several correction +turns. It stores the final domain artifact, not a raw intermediate response, +`RunResult`, exchange, or prepared handle. Failed, rejected, and +validation-incomplete work is not saved. A hit requires ordinary decoding and +usability checks; it does not replay model work or claim fresh domain validation. + +## Ownership And Scope + +PromptKit owns everything within a native request execution, including structural +response validation, transport retries, and internal repair attempts. Notarius +consumes the public response or terminal failure and then owns domain acceptance. +Those internal attempts are not inputs to the cache's eligibility policy. + +This request is solely for reusable message-content hashing. It does not request +PromptKit-owned cache storage, a cache-hit execution result, cached-response +injection, validation replay, exchange restoration, or a write-through response +cache. Debug observations do not change this boundary. Notarius owns cache +lookup, final acceptance, persistence, and reuse/bypass/refresh controls. + +## Suggested Verification + +- Repeated preparation in separate engines produces the same hash for identical + role/text sequences despite different preparation timing or session identity. +- Changing a role, text (including whitespace), order, message count, or boundary + changes the hash; ambiguous concatenations do not collapse to the same input. +- Settings-only, prompt-ID-only, and cache-hint-only changes preserve the hash + when the effective role/text sequence is unchanged. +- Rendered inputs, appended messages, and captured history participate in the + hash when they contribute to the effective messages. +- The hash is available before model execution, and reading it does not consume + the prepared handle or alter its retained snapshot. +- Existing `RenderedPromptHash` behavior remains intact. Document the new value + in public GoDoc and add a short consumer example of lookup before execution. + +Notarius will verify its own cache behavior separately: hits make no model calls, +only accepted artifacts are saved, and corrected answers remain keyed to the +initial task. The supplying PromptKit release will be selected during Notarius +roadmap incorporation; this request does not assign a release number.