319 lines
17 KiB
Markdown
319 lines
17 KiB
Markdown
# Sequence-Item Placement and Identity-Based Composition
|
|
|
|
Status: design decisions settled; not implemented. This roadmap owns the target
|
|
state and accepted behavior. The [implementation plan](implementation.md) owns
|
|
concrete API/syntax choices, execution order, validation, and progress.
|
|
|
|
## Goal and current baseline
|
|
|
|
Let consumers assemble a list of records from independently authored mapping
|
|
files. Placement makes a source mapping one sequence item; identity-based strict
|
|
composition combines distinct items and rejects repeated identities. These are
|
|
complementary operations, with explicit consumer opt-in.
|
|
|
|
Current [input destinations](../config.md#input-destinations) place source
|
|
mappings beneath mapping-key paths. Imports support immediate-parent-relative
|
|
destinations. Current [strict composition](../config.md#strict-document-composition)
|
|
combines mappings recursively and rejects other overlaps, while
|
|
[precedence assembly](../config.md#layered-loading) and
|
|
[selected overlays](../config.md#selected-overlays) replace sequences wholesale.
|
|
Neither sequence-item placement nor sequence-composition registration exists today.
|
|
|
|
This roadmap covers a narrower feature than the
|
|
[identified record sequences candidate](future.md#1-identified-record-sequences).
|
|
Composite identities, recursive merging of matching records, identity-based
|
|
overlays, and record inheritance are not requested here. That scope statement
|
|
does not establish permanent architectural prohibitions.
|
|
|
|
## Targeted behavior
|
|
|
|
### PromptKit use case
|
|
|
|
Each file defines one prompt directly.
|
|
|
|
`prompt_one.yml`:
|
|
|
|
```yaml
|
|
id: prompt.one
|
|
text: "foo"
|
|
```
|
|
|
|
`prompt_two.yml`:
|
|
|
|
```yaml
|
|
id: prompt.two
|
|
text: "bar"
|
|
```
|
|
|
|
The consumer selects item placement at `prompts` for both sources and registers
|
|
`id` as that sequence's identity field. Within one strict composition tier, the
|
|
result is:
|
|
|
|
```yaml
|
|
prompts:
|
|
- id: prompt.one
|
|
text: "foo"
|
|
- id: prompt.two
|
|
text: "bar"
|
|
```
|
|
|
|
The files need neither a filename-derived wrapper key nor their own list wrapper.
|
|
Existing sources that already contain `prompts: [...]` can contribute to the same
|
|
registered sequence under the same composition rule.
|
|
|
|
### Explicit sequence-item placement
|
|
|
|
- The destination identifies where content belongs; an explicit placement mode
|
|
selects ordinary mapping placement or placement as one sequence item. Preserve
|
|
existing behavior when the new mode is not selected. Do not infer the mode
|
|
from document shape, filename, or the presence of an `id` field.
|
|
- Support item placement for application-supplied inputs and YAML path/glob
|
|
imports. Each matched file of an item-placement glob contributes one item,
|
|
using existing match and traversal ordering. Physical import-path resolution
|
|
remains relative to the declaring file.
|
|
- Wrap the source's application mapping as one item at the destination sequence;
|
|
the new operation is not flattening a source sequence or splitting a mapping
|
|
into several items. Existing supported source forms should receive equivalent
|
|
placement behavior; this does not expand accepted source-root shapes.
|
|
- Item placement supplies structure, not identity values. Keep authored `id` and
|
|
other fields in the item. Source directives and version headers retain their
|
|
own processing rules, with the item-placement interpretation recorded under
|
|
Settled Decisions.
|
|
- Preserve existing absence distinctions: `NoSource()` and an unmatched optional
|
|
import contribute nothing. An actual empty mapping contributes an empty item,
|
|
which fails the agreed identity requirement when that item participates in a
|
|
registered composition sequence.
|
|
|
|
Item placement is independent of composition registration. Ordinary imports
|
|
inside an item compose into its mapping; public source evidence reports the
|
|
placement mode and destination container. The settled decisions below define
|
|
these behaviors in detail.
|
|
|
|
### Identity-based strict composition
|
|
|
|
- Consumers designate sequence paths and one direct string identity field for
|
|
each. Unregistered sequences retain their current composition behavior.
|
|
- Each participating record is a mapping with the designated string field.
|
|
Missing, null, and non-string identities are invalid. Compare exact strings
|
|
without trimming, case folding, normalization, or numeric coercion. Empty
|
|
strings are valid identities; repeated empty identities conflict.
|
|
- Append distinct records in contribution order, preserving each source list's
|
|
order. Use existing deterministic traversal within a strict tier rather than
|
|
sorting by identity.
|
|
- Repeated identities conflict across contributions and within one contributing
|
|
list, including a list with no peer contribution. Equal bodies and disjoint
|
|
bodies both conflict; matching records do not recursively merge.
|
|
- An empty sequence contributes no records. Two empty sequences compose, and an
|
|
explicitly contributed empty sequence remains present.
|
|
- Apply composition to main inputs and their imports and within each underlay
|
|
group, including placed item contributions and already-authored sequences.
|
|
Identity uniqueness is scoped to the sequence in that strict tier.
|
|
- A composition error cannot be hidden by a higher-precedence replacement. Use
|
|
normal load errors and the existing failed-load result contract.
|
|
|
|
This differs from mapping composition at a shared key: mapping values can recurse,
|
|
whereas a repeated record identity conflicts. It does not establish final
|
|
uniqueness after defaults or application resolution. Identity values are not
|
|
supplied by those later phases; final domain validation remains application-owned.
|
|
|
|
### Precedence and other operations
|
|
|
|
Between underlay tiers, between underlays and the main tier, and in selected
|
|
overlays, sequences continue to replace lower sequences wholesale. `[]` clears
|
|
the lower list. The same identity can occur in different tiers without a
|
|
cross-tier duplicate error. Item placement does not turn precedence assembly
|
|
into append semantics.
|
|
|
|
Whole-sequence CLI overrides, defaults, application resolution, and existing
|
|
sibling inheritance retain their behavior. This feature does not make records
|
|
inherit from one another by identity. Effective-configuration diffs continue to
|
|
compare final sequences atomically. Replacement-only selected overlays and
|
|
synthetic CLI overrides do not acquire identity validation from composition
|
|
registration; existing decoding and application validation still apply.
|
|
|
|
### Existing inspection and ownership obligations
|
|
|
|
A field at source path `text` can become `prompts[1].text`; a record originally
|
|
at source index zero can acquire a different effective index after composition.
|
|
Keep original physical locations and resource contexts associated with the right
|
|
record while reflecting effective placement in presence and origin queries.
|
|
|
|
History and explanations must represent accumulation within a tier and actual
|
|
replacement between tiers. Replaying each contribution as a replacement would
|
|
lose earlier records. Apply existing historical sensitivity and disclosure rules
|
|
when indexes change or sequences are replaced, and preserve detached public
|
|
records and existing input/result ownership. These are applications of current
|
|
contracts, not a new promise of persistent identity-based paths across loads.
|
|
|
|
## Work needed
|
|
|
|
The target state requires changes in these areas; execution order belongs to the
|
|
[implementation plan](implementation.md):
|
|
|
|
- **Public API and import syntax:** introduce the selected composition
|
|
registration and placement controls at the existing root-owned boundaries;
|
|
convert them into private engine state. Define invalid-option handling and
|
|
retain current defaults for callers that do not opt in.
|
|
- **Acquisition and placement:** support item contributions in ordinary and
|
|
layered loading, input/import destinations, and glob expansion. Implement the
|
|
selected nested-import and source-version interpretation without changing
|
|
acquisition identity or physical resource resolution.
|
|
- **Composition and retained evidence:** extend claim validation, tree assembly,
|
|
origins, and history together. The current composer in
|
|
`internal/engine/compose.go` collects claims by path and stores history
|
|
applications as well as the final tree. Updating only the tree merge is
|
|
insufficient. Keep tier boundaries explicit and preserve the existing limits
|
|
and accounting through the new wrapper and accumulation operations.
|
|
- **Inspection integration:** carry source-to-effective placement through history,
|
|
explanations, sensitivity, resource contexts, and public source records. Use
|
|
the smallest internal representation that preserves those existing contracts;
|
|
this roadmap does not prescribe a new indexing or caching subsystem.
|
|
- **Behavioral verification:** cover direct and glob-imported item sources,
|
|
already-authored lists, mixed contributions, duplicate/malformed identities,
|
|
empty and absent contributions, ordering, mounted paths, and nested imports.
|
|
Verify whole-list replacement, errors hidden by later tiers, version and
|
|
structural processing, remapped indexes, resource contexts, historical
|
|
redaction, and detached records. Protect current unregistered behavior,
|
|
defaults/resolution, and atomic diffs with proportionate regression coverage.
|
|
- **Documentation and adoption:** provide a maintained PromptKit-shaped example
|
|
and update public declaration comments, configuration/inspection contracts,
|
|
consumer guidance, and relevant internal documentation when behavior lands.
|
|
Record the significant design decision under the architecture policy and keep
|
|
the broader candidate in `future.md` distinct from the implemented subset.
|
|
|
|
Completion means the PromptKit example works through public inputs and imports,
|
|
composition and replacement behave as agreed, and configuration values agree
|
|
with their inspection evidence. Validation follows the
|
|
[development guide](../development.md#baseline-validation).
|
|
|
|
## Settled Decisions
|
|
|
|
The following choices are accepted for this feature. No substantive design
|
|
choices remain open in this roadmap. The implementation plan selects exact
|
|
public names, YAML option names, and existing diagnostic-code mappings.
|
|
Internal representations remain implementation choices, subject to the existing
|
|
contracts and the behavior recorded here.
|
|
|
|
### 1. Public controls and ownership
|
|
|
|
Put sequence path/identity-field registrations on `Definition`. Put an explicit
|
|
placement mode beside input destination controls, with equivalent YAML import
|
|
syntax and support in ordinary loading. This separates reusable composition
|
|
policy from placement of individual sources. Default placement remains unchanged.
|
|
|
|
### 2. Destination and registration paths
|
|
|
|
Use literal mapping-key paths to identify destination sequences and match
|
|
registrations at their effective location after placement. For example, a source
|
|
mounted beneath `application` uses a registration at `application.prompts`.
|
|
Preserve existing destination validation without adding new reserved names.
|
|
|
|
Permit unused registrations, coalesce identical registrations, and reject
|
|
conflicting identity fields for the same path. Ancestor/descendant registrations
|
|
apply only where their literal paths reach sequences; they do not implicitly
|
|
traverse every record.
|
|
|
|
Sequence-index selectors and wildcard matching are outside this initial feature.
|
|
Item placement identifies a sequence beneath the configuration root; a root
|
|
sequence would expand the existing mapping-root model and is not included.
|
|
These are the selected feature boundaries, not permanent architectural exclusions.
|
|
Nested placement follows decision 4 without expanding registration addressing.
|
|
|
|
### 3. Item placement without identity registration
|
|
|
|
Allow item placement independently. It contributes a singleton sequence without
|
|
requiring an identity registration. Existing rules apply at unregistered paths:
|
|
two present sequences conflict in one strict tier, and an upper-tier sequence
|
|
replaces a lower one. Do not infer append semantics merely from item placement.
|
|
|
|
### 4. Imports declared inside an item-placed source
|
|
|
|
Compose ordinary imports into the declaring item's mapping. For an import
|
|
declared at the item root, an omitted destination or `destination: []` means that
|
|
item context; `[metadata]` places the helper inside the item's `metadata`.
|
|
Further relative destinations continue from the immediately importing document's
|
|
placement within that context. Physical import paths still resolve from the
|
|
declaring file.
|
|
|
|
Item mode is explicit on each import, not inherited automatically. A collection
|
|
manifest explicitly imports peer prompt files as items at `prompts`. A helper
|
|
import supplies fields of the declaring item and need not declare its own ID.
|
|
|
|
An explicitly item-placed import inside a prompt can contribute to a nested
|
|
sequence. Registrations do not reach through outer sequence indexes in this
|
|
initial feature, so those nested sequences retain atomic composition. Multiple
|
|
contributions to such a nested sequence therefore follow ordinary strict
|
|
conflict rules. This permits nested placement without introducing index selectors
|
|
or wildcard matching.
|
|
|
|
### 5. Identity-validation boundary and structural processing
|
|
|
|
Validate each completed record after applicable source-header processing and
|
|
ordinary helper-import composition, before combining records in the collection
|
|
and before later precedence replacement. A helper may supply the identity field;
|
|
it need not be authored in the item file itself. Duplicate fields within the
|
|
item still follow ordinary strict mapping composition. Defaults and application
|
|
resolution cannot supply missing identities because they run later.
|
|
|
|
Preserve existing structural semantics. An item has no mapping-sibling namespace
|
|
for record inheritance, while mappings inside it can use existing inheritance.
|
|
Consumed structural fields and headers do not become application identity
|
|
values. Existing processing rules govern overlap with those controls; no new
|
|
identity-field-name blacklist is introduced.
|
|
|
|
### 6. Empty identities and literal field names
|
|
|
|
Accept empty string identities; repeated empty identities conflict. Interpret
|
|
the designated direct field name literally, including unusual or empty YAML
|
|
mapping keys where existing processing permits them. Presence and string type
|
|
suffice for identity comparison. Nonempty-ID requirements, naming conventions,
|
|
and whitespace restrictions remain application-owned validation choices.
|
|
|
|
### 7. Non-sequence values at registered paths
|
|
|
|
Registration selects how actual sequences compose; it is not an additional
|
|
schema declaration. Missing and non-sequence values retain existing composition
|
|
and decoding behavior. Sequence/null and sequence/mapping overlaps still
|
|
conflict in strict composition. Item placement contributes an actual sequence
|
|
and follows these rules. Registration does not independently reject every
|
|
non-sequence source value that a later tier might replace.
|
|
|
|
### 8. Validation of replacement-only sequences
|
|
|
|
Apply identity validation to registered sequences participating in strict
|
|
composition, including singleton contributions. Selected-overlay sequences and
|
|
synthetic CLI replacements retain their existing validation. Defaults and
|
|
application callbacks also retain their behavior.
|
|
|
|
An overlay can therefore replace a list with duplicate IDs unless the consuming
|
|
application's validator rejects them. The identity rule governs composition,
|
|
not final configuration validity.
|
|
|
|
### 9. Source-version checks for item-placed main inputs
|
|
|
|
Support item placement for main inputs as well as imported and underlay sources.
|
|
Enabled source-version modes inspect declarations in source-local coordinates
|
|
before placement. Default mode retains its existing effective-path interpretation.
|
|
Accepted non-main headers are consumed; the main header remains application
|
|
content, following existing rules.
|
|
|
|
The retained main header's effective location includes its actual item index.
|
|
Existing ownership protection follows that location and must not accidentally
|
|
protect another record because its source index differs from its effective
|
|
index. Preserve source-local version evidence and diagnostics where the existing
|
|
contract uses them. This feature introduces neither a new version policy nor
|
|
an exception to existing version-path protections.
|
|
|
|
### 10. Public placement evidence and diagnostics
|
|
|
|
Keep `SourceInfo.Destination` describing the effective destination container and
|
|
expose the placement mode. Use existing effective-path metadata to describe
|
|
resulting item paths, retaining physical source locations and resource contexts.
|
|
Do not promise one permanent item index per source: replacement can remove its
|
|
contribution, and contribution order can differ between loads.
|
|
|
|
Duplicate diagnostics identify conflicting locations and the sequence path
|
|
without exposing potentially sensitive identity values. Reuse existing diagnostic
|
|
categories where appropriate. Resolved historical item paths and persistent
|
|
identity-addressed paths in source records are not part of this feature.
|