diff options
| author | Adam Malczewski <[email protected]> | 2026-06-27 04:18:59 +0900 |
|---|---|---|
| committer | Adam Malczewski <[email protected]> | 2026-06-27 04:18:59 +0900 |
| commit | 3566a20ebbded754070fce66af48690d1a904879 (patch) | |
| tree | ad4941abdbe685e2d4ae85d9ad3d0b6e9cf82c2d /.dispatch | |
| parent | 2fa03f8d7410c2b8d6be8e10ad088863e83d7177 (diff) | |
| download | dispatch-web-3566a20ebbded754070fce66af48690d1a904879.tar.gz dispatch-web-3566a20ebbded754070fce66af48690d1a904879.zip | |
feat(vision): image paste + transcript image rendering + vision badge
Vision & vision-handoff frontend (consumes the backend's additive
[email protected] / [email protected] image types — no version bump).
Contracts mirrored:
- .dispatch/wire.reference.md: ImageChunk added to the Chunk union +
ImageChunk/ImageInput interfaces.
- .dispatch/transport-contract.reference.md: ChatRequest.images,
ModelMetadata.vision, + ImageChunk/ImageInput re-exports.
Core (core/chunks):
- conformance: assertChunkExhaustive handles the new 'image' variant
(the guard caught it — its purpose).
- appendUserMessage(state, text, images?) echoes a [text, image, ...]
user run; the user-message event dedup scans the trailing user run
(not just the last chunk) so an image-bearing echo doesn't duplicate
the text; applyHistory's during-gen dedup matches a multi-chunk echo
by content equality (chunkContentEquals + trailingRun helpers).
UI:
- ChatView renders user 'image' chunks as lazy <img> bubbles; a
non-vision model's persisted [image, analysis-text] both render.
read_image tool renders generically (no special-casing).
- Composer: clipboard paste / file picker / drag-drop of images ->
base64 data URLs, thumbnail previews with remove, forwarded on
chat.send (omitted when none). Image-only sends allowed; steering
(chat.queue) never forwards images.
- ModelSelector: vision badge (isVisionModel) marks vision-capable
models; indicator shows native-vision vs vision-handoff hint.
Store wiring: ChatStore.send + AppStore.send + App.svelte handleSend
thread images through; chat.send still omits cwd (only images added).
Verification: svelte-check 0/0; vitest 901/901 (run twice, +34 new);
biome clean; vite build OK. See backend-handoff.md §2j.
Not merged or pushed.
Diffstat (limited to '.dispatch')
| -rw-r--r-- | .dispatch/transport-contract.reference.md | 39 | ||||
| -rw-r--r-- | .dispatch/wire.reference.md | 51 |
2 files changed, 89 insertions, 1 deletions
diff --git a/.dispatch/transport-contract.reference.md b/.dispatch/transport-contract.reference.md index 48ba45d..152a1b0 100644 --- a/.dispatch/transport-contract.reference.md +++ b/.dispatch/transport-contract.reference.md @@ -8,6 +8,15 @@ > **Orchestrator:** SNAPSHOT of `[email protected]` (MCP status + computers). Regenerate whenever > it changes. > +> **2026-06-26 delta (vision handoff — ADDITIVE, NO version bump):** adds the vision/image surface. +> `ChatRequest` (+ `ChatSendMessage`/`QueueRequest`) gains an optional `images?: readonly ImageInput[]` +> (each entry: `{ url, mimeType? }` — a base64 data URL or `http(s)://` URL; validated non-array/no-url/ +> empty-url → 400, empty array treated as absent). `ModelMetadata` gains `vision?: boolean` (true when the +> model natively accepts images; absent → the server's vision handoff transcribes images to text before the +> model sees them). `ImageChunk`/`ImageInput` are `@dispatch/wire` types (re-exported here). A non-vision +> model's persisted user message keeps the original `image` chunk AND adds a `text` transcription chunk +> (`[Image analysis (via <model>)]: …`) in the SAME message — render both. See `backend-handoff.md` §2j. +> > **2026-06-25 delta (SSH handoff #2 — ADDITIVE to `[email protected]`, NO version bump):** adds the > computer HTTP API types: `ComputerListResponse` (`GET /computers`), `ComputerResponse` (`GET /computers/:alias`), > `ComputerStatusResponse` (`GET /computers/:alias/status`), `TestComputerResponse` (`POST /computers/:alias/test`), @@ -59,8 +68,11 @@ import type { SurfaceClientMessage, SurfaceServerMessage } from "@dispatch/ui-contract"; import type { AgentEvent, + Computer, + ComputerEntry, ConversationMeta, ConversationStatus, + ImageInput, QueuedMessage, ReasoningEffort, StoredChunk, @@ -72,8 +84,12 @@ import type { export type { AgentEvent, CompactionResult, + Computer, + ComputerEntry, ConversationMeta, ConversationStatus, + ImageChunk, + ImageInput, QueuedMessage, ReasoningEffort, StepMetrics, @@ -101,6 +117,21 @@ export interface ChatRequest { readonly message: string; /** + * Images attached to this turn (e.g. a user-pasted screenshot). Each entry's + * `url` is a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` + * URL. The server converts these to `image` chunks on the persisted user + * message. For a VISION-capable model (e.g. kimi), the images are passed + * through to the provider natively. For a NON-vision model (e.g. glm-5.2), + * the server's vision handoff transcribes each image to a text description + * (via a vision-capable model) and feeds that text instead — so a text-only + * model can still reason about the image's contents. Optional — omit for a + * text-only turn (backward compatible). Validation: non-array `images` → + * 400; an image without `url` → 400; empty `url` → 400. An empty array is + * accepted and treated as absent. + */ + readonly images?: readonly ImageInput[]; + + /** * The model to use, as a model name in `<credentialName>/<model>` form — one * of the exact strings returned by `GET /models`. Omit to use the server's * default credential + model. @@ -157,6 +188,14 @@ export interface ModelsResponse { /** Per-model metadata returned alongside the model catalog. */ export interface ModelMetadata { readonly contextWindow?: number; + /** + * Whether this model can natively accept image input (vision/multimodal). + * When `true`, image chunks in a user message are passed through to the + * provider. When `false`/absent, the server's vision handoff transcribes + * images to text before the model sees them. A client may use this to show + * a vision badge in the model picker. Optional — absent when unknown. + */ + readonly vision?: boolean; } /** diff --git a/.dispatch/wire.reference.md b/.dispatch/wire.reference.md index 05ed40b..c40fe81 100644 --- a/.dispatch/wire.reference.md +++ b/.dispatch/wire.reference.md @@ -6,6 +6,14 @@ > > **Orchestrator:** SNAPSHOT of `[email protected]` (workspaces + computers). Regenerate whenever `@dispatch/wire` changes. > +> **2026-06-26 delta (vision handoff — ADDITIVE to `[email protected]`, NO version bump):** adds a new +> `ImageChunk` variant to the `Chunk` union (`{ type: "image", url, mimeType? }` — `url` is a base64 data +> URL or an `http(s)://` URL) and a transport-facing `ImageInput` (`{ url, mimeType? }`, what a client +> sends on `ChatRequest.images`; the orchestrator converts each into an `ImageChunk` on the persisted user +> message). Vision-capable models receive image chunks natively; non-vision models never see them directly +> — the orchestrator's vision handoff transcribes each to a text description (persisted as a separate +> `text` chunk in the SAME user message). See `backend-handoff.md` §2j. +> > **2026-06-23 delta (workspaces handoff — package bumped `0.11.0` → `0.12.0`, ADDITIVE):** adds > `Workspace` + `WorkspaceEntry` (a list entry with a conversation count) and a required > `workspaceId: string` on `ConversationMeta` (`"default"` for legacy/unspecified conversations). A @@ -68,7 +76,8 @@ export type Chunk = | ToolCallChunk | ToolResultChunk | ErrorChunk - | SystemChunk; + | SystemChunk + | ImageChunk; /** A piece of plain text content from the assistant or user. */ export interface TextChunk { @@ -145,6 +154,46 @@ export interface SystemChunk { } /** + * An image attached to a message (e.g. a user-pasted screenshot or pasted + * photo). Carries a `url` that is EITHER a base64 data URL + * (`data:image/png;base64,…`) OR an `http(s)://` URL. Vision-capable models + * receive it natively (the provider serializes it to its image-content + * format); non-vision models never see it directly — the orchestrator's + * **vision handoff** transcribes it to a text description (via a + * vision-capable model) and feeds that text instead, so a text-only model can + * still reason about the image's contents. + * + * When a transcription was performed, it is persisted as a separate `text` + * chunk alongside the `image` chunk in the SAME user message, so the + * description is reused on every later turn (no re-transcription) and a + * client renders both the original image and its textual analysis. + */ +export interface ImageChunk { + readonly type: "image"; + /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */ + readonly url: string; + /** + * Optional MIME type of the image (e.g. `"image/png"`). Inferred from the + * data URL when absent; present so a client can render an icon/label without + * parsing the URL. Optional — callers that only have a URL omit it. + */ + readonly mimeType?: string; +} + +/** + * An image a client attaches to a chat message (`ChatRequest.images`). The + * transport-facing input shape; the orchestrator converts each `ImageInput` + * into an `ImageChunk` on the persisted user message. Carries the same `url` + * semantics as `ImageChunk.url`. + */ +export interface ImageInput { + /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */ + readonly url: string; + /** Optional MIME type (e.g. `"image/png"`). Optional — inferred from the data URL when absent. */ + readonly mimeType?: string; +} + +/** * A chat message: a role plus an ordered sequence of chunks. Messages are the * unit passed to and from the provider; chunks are the unit persisted and * rendered. |
