summaryrefslogtreecommitdiffhomepage
path: root/.dispatch/wire.reference.md
diff options
context:
space:
mode:
authorAdam Malczewski <[email protected]>2026-06-27 04:18:59 +0900
committerAdam Malczewski <[email protected]>2026-06-27 04:18:59 +0900
commit3566a20ebbded754070fce66af48690d1a904879 (patch)
treead4941abdbe685e2d4ae85d9ad3d0b6e9cf82c2d /.dispatch/wire.reference.md
parent2fa03f8d7410c2b8d6be8e10ad088863e83d7177 (diff)
downloaddispatch-web-3566a20ebbded754070fce66af48690d1a904879.tar.gz
dispatch-web-3566a20ebbded754070fce66af48690d1a904879.zip
feat(vision): image paste + transcript image rendering + vision badge
Vision & vision-handoff frontend (consumes the backend's additive [email protected] / [email protected] image types — no version bump). Contracts mirrored: - .dispatch/wire.reference.md: ImageChunk added to the Chunk union + ImageChunk/ImageInput interfaces. - .dispatch/transport-contract.reference.md: ChatRequest.images, ModelMetadata.vision, + ImageChunk/ImageInput re-exports. Core (core/chunks): - conformance: assertChunkExhaustive handles the new 'image' variant (the guard caught it — its purpose). - appendUserMessage(state, text, images?) echoes a [text, image, ...] user run; the user-message event dedup scans the trailing user run (not just the last chunk) so an image-bearing echo doesn't duplicate the text; applyHistory's during-gen dedup matches a multi-chunk echo by content equality (chunkContentEquals + trailingRun helpers). UI: - ChatView renders user 'image' chunks as lazy <img> bubbles; a non-vision model's persisted [image, analysis-text] both render. read_image tool renders generically (no special-casing). - Composer: clipboard paste / file picker / drag-drop of images -> base64 data URLs, thumbnail previews with remove, forwarded on chat.send (omitted when none). Image-only sends allowed; steering (chat.queue) never forwards images. - ModelSelector: vision badge (isVisionModel) marks vision-capable models; indicator shows native-vision vs vision-handoff hint. Store wiring: ChatStore.send + AppStore.send + App.svelte handleSend thread images through; chat.send still omits cwd (only images added). Verification: svelte-check 0/0; vitest 901/901 (run twice, +34 new); biome clean; vite build OK. See backend-handoff.md §2j. Not merged or pushed.
Diffstat (limited to '.dispatch/wire.reference.md')
-rw-r--r--.dispatch/wire.reference.md51
1 files changed, 50 insertions, 1 deletions
diff --git a/.dispatch/wire.reference.md b/.dispatch/wire.reference.md
index 05ed40b..c40fe81 100644
--- a/.dispatch/wire.reference.md
+++ b/.dispatch/wire.reference.md
@@ -6,6 +6,14 @@
>
> **Orchestrator:** SNAPSHOT of `[email protected]` (workspaces + computers). Regenerate whenever `@dispatch/wire` changes.
>
+> **2026-06-26 delta (vision handoff — ADDITIVE to `[email protected]`, NO version bump):** adds a new
+> `ImageChunk` variant to the `Chunk` union (`{ type: "image", url, mimeType? }` — `url` is a base64 data
+> URL or an `http(s)://` URL) and a transport-facing `ImageInput` (`{ url, mimeType? }`, what a client
+> sends on `ChatRequest.images`; the orchestrator converts each into an `ImageChunk` on the persisted user
+> message). Vision-capable models receive image chunks natively; non-vision models never see them directly
+> — the orchestrator's vision handoff transcribes each to a text description (persisted as a separate
+> `text` chunk in the SAME user message). See `backend-handoff.md` §2j.
+>
> **2026-06-23 delta (workspaces handoff — package bumped `0.11.0` → `0.12.0`, ADDITIVE):** adds
> `Workspace` + `WorkspaceEntry` (a list entry with a conversation count) and a required
> `workspaceId: string` on `ConversationMeta` (`"default"` for legacy/unspecified conversations). A
@@ -68,7 +76,8 @@ export type Chunk =
| ToolCallChunk
| ToolResultChunk
| ErrorChunk
- | SystemChunk;
+ | SystemChunk
+ | ImageChunk;
/** A piece of plain text content from the assistant or user. */
export interface TextChunk {
@@ -145,6 +154,46 @@ export interface SystemChunk {
}
/**
+ * An image attached to a message (e.g. a user-pasted screenshot or pasted
+ * photo). Carries a `url` that is EITHER a base64 data URL
+ * (`data:image/png;base64,…`) OR an `http(s)://` URL. Vision-capable models
+ * receive it natively (the provider serializes it to its image-content
+ * format); non-vision models never see it directly — the orchestrator's
+ * **vision handoff** transcribes it to a text description (via a
+ * vision-capable model) and feeds that text instead, so a text-only model can
+ * still reason about the image's contents.
+ *
+ * When a transcription was performed, it is persisted as a separate `text`
+ * chunk alongside the `image` chunk in the SAME user message, so the
+ * description is reused on every later turn (no re-transcription) and a
+ * client renders both the original image and its textual analysis.
+ */
+export interface ImageChunk {
+ readonly type: "image";
+ /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */
+ readonly url: string;
+ /**
+ * Optional MIME type of the image (e.g. `"image/png"`). Inferred from the
+ * data URL when absent; present so a client can render an icon/label without
+ * parsing the URL. Optional — callers that only have a URL omit it.
+ */
+ readonly mimeType?: string;
+}
+
+/**
+ * An image a client attaches to a chat message (`ChatRequest.images`). The
+ * transport-facing input shape; the orchestrator converts each `ImageInput`
+ * into an `ImageChunk` on the persisted user message. Carries the same `url`
+ * semantics as `ImageChunk.url`.
+ */
+export interface ImageInput {
+ /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */
+ readonly url: string;
+ /** Optional MIME type (e.g. `"image/png"`). Optional — inferred from the data URL when absent. */
+ readonly mimeType?: string;
+}
+
+/**
* A chat message: a role plus an ordered sequence of chunks. Messages are the
* unit passed to and from the provider; chunks are the unit persisted and
* rendered.