From 3566a20ebbded754070fce66af48690d1a904879 Mon Sep 17 00:00:00 2001 From: Adam Malczewski Date: Sat, 27 Jun 2026 04:18:59 +0900 Subject: feat(vision): image paste + transcript image rendering + vision badge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vision & vision-handoff frontend (consumes the backend's additive wire@0.12.0 / transport-contract@0.22.0 image types — no version bump). Contracts mirrored: - .dispatch/wire.reference.md: ImageChunk added to the Chunk union + ImageChunk/ImageInput interfaces. - .dispatch/transport-contract.reference.md: ChatRequest.images, ModelMetadata.vision, + ImageChunk/ImageInput re-exports. Core (core/chunks): - conformance: assertChunkExhaustive handles the new 'image' variant (the guard caught it — its purpose). - appendUserMessage(state, text, images?) echoes a [text, image, ...] user run; the user-message event dedup scans the trailing user run (not just the last chunk) so an image-bearing echo doesn't duplicate the text; applyHistory's during-gen dedup matches a multi-chunk echo by content equality (chunkContentEquals + trailingRun helpers). UI: - ChatView renders user 'image' chunks as lazy bubbles; a non-vision model's persisted [image, analysis-text] both render. read_image tool renders generically (no special-casing). - Composer: clipboard paste / file picker / drag-drop of images -> base64 data URLs, thumbnail previews with remove, forwarded on chat.send (omitted when none). Image-only sends allowed; steering (chat.queue) never forwards images. - ModelSelector: vision badge (isVisionModel) marks vision-capable models; indicator shows native-vision vs vision-handoff hint. Store wiring: ChatStore.send + AppStore.send + App.svelte handleSend thread images through; chat.send still omits cwd (only images added). Verification: svelte-check 0/0; vitest 901/901 (run twice, +34 new); biome clean; vite build OK. See backend-handoff.md §2j. Not merged or pushed. --- .dispatch/transport-contract.reference.md | 39 +++++ .dispatch/wire.reference.md | 51 +++++- GLOSSARY.md | 3 + backend-handoff.md | 89 +++++++++- src/app/App.svelte | 13 +- src/app/store.svelte.ts | 17 +- src/app/store.test.ts | 28 +++ src/core/chunks/reducer.test.ts | 153 +++++++++++++++++ src/core/chunks/reducer.ts | 153 ++++++++++++++--- src/core/wire/conformance.test.ts | 40 ++++- src/core/wire/conformance.ts | 2 + src/features/chat/model-select.test.ts | 33 +++- src/features/chat/model-select.ts | 15 ++ src/features/chat/store.svelte.ts | 16 +- src/features/chat/store.test.ts | 89 +++++++++- src/features/chat/ui.test.ts | 272 +++++++++++++++++++++++++++++- src/features/chat/ui/ChatView.svelte | 12 +- src/features/chat/ui/Composer.svelte | 232 +++++++++++++++++++++++-- src/features/chat/ui/ModelSelector.svelte | 46 ++++- 19 files changed, 1230 insertions(+), 73 deletions(-) diff --git a/.dispatch/transport-contract.reference.md b/.dispatch/transport-contract.reference.md index 48ba45d..152a1b0 100644 --- a/.dispatch/transport-contract.reference.md +++ b/.dispatch/transport-contract.reference.md @@ -8,6 +8,15 @@ > **Orchestrator:** SNAPSHOT of `transport-contract@0.22.0` (MCP status + computers). Regenerate whenever > it changes. > +> **2026-06-26 delta (vision handoff — ADDITIVE, NO version bump):** adds the vision/image surface. +> `ChatRequest` (+ `ChatSendMessage`/`QueueRequest`) gains an optional `images?: readonly ImageInput[]` +> (each entry: `{ url, mimeType? }` — a base64 data URL or `http(s)://` URL; validated non-array/no-url/ +> empty-url → 400, empty array treated as absent). `ModelMetadata` gains `vision?: boolean` (true when the +> model natively accepts images; absent → the server's vision handoff transcribes images to text before the +> model sees them). `ImageChunk`/`ImageInput` are `@dispatch/wire` types (re-exported here). A non-vision +> model's persisted user message keeps the original `image` chunk AND adds a `text` transcription chunk +> (`[Image analysis (via )]: …`) in the SAME message — render both. See `backend-handoff.md` §2j. +> > **2026-06-25 delta (SSH handoff #2 — ADDITIVE to `transport-contract@0.22.0`, NO version bump):** adds the > computer HTTP API types: `ComputerListResponse` (`GET /computers`), `ComputerResponse` (`GET /computers/:alias`), > `ComputerStatusResponse` (`GET /computers/:alias/status`), `TestComputerResponse` (`POST /computers/:alias/test`), @@ -59,8 +68,11 @@ import type { SurfaceClientMessage, SurfaceServerMessage } from "@dispatch/ui-contract"; import type { AgentEvent, + Computer, + ComputerEntry, ConversationMeta, ConversationStatus, + ImageInput, QueuedMessage, ReasoningEffort, StoredChunk, @@ -72,8 +84,12 @@ import type { export type { AgentEvent, CompactionResult, + Computer, + ComputerEntry, ConversationMeta, ConversationStatus, + ImageChunk, + ImageInput, QueuedMessage, ReasoningEffort, StepMetrics, @@ -100,6 +116,21 @@ export interface ChatRequest { /** The user's message text for this turn. */ readonly message: string; + /** + * Images attached to this turn (e.g. a user-pasted screenshot). Each entry's + * `url` is a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` + * URL. The server converts these to `image` chunks on the persisted user + * message. For a VISION-capable model (e.g. kimi), the images are passed + * through to the provider natively. For a NON-vision model (e.g. glm-5.2), + * the server's vision handoff transcribes each image to a text description + * (via a vision-capable model) and feeds that text instead — so a text-only + * model can still reason about the image's contents. Optional — omit for a + * text-only turn (backward compatible). Validation: non-array `images` → + * 400; an image without `url` → 400; empty `url` → 400. An empty array is + * accepted and treated as absent. + */ + readonly images?: readonly ImageInput[]; + /** * The model to use, as a model name in `/` form — one * of the exact strings returned by `GET /models`. Omit to use the server's @@ -157,6 +188,14 @@ export interface ModelsResponse { /** Per-model metadata returned alongside the model catalog. */ export interface ModelMetadata { readonly contextWindow?: number; + /** + * Whether this model can natively accept image input (vision/multimodal). + * When `true`, image chunks in a user message are passed through to the + * provider. When `false`/absent, the server's vision handoff transcribes + * images to text before the model sees them. A client may use this to show + * a vision badge in the model picker. Optional — absent when unknown. + */ + readonly vision?: boolean; } /** diff --git a/.dispatch/wire.reference.md b/.dispatch/wire.reference.md index 05ed40b..c40fe81 100644 --- a/.dispatch/wire.reference.md +++ b/.dispatch/wire.reference.md @@ -6,6 +6,14 @@ > > **Orchestrator:** SNAPSHOT of `wire@0.12.0` (workspaces + computers). Regenerate whenever `@dispatch/wire` changes. > +> **2026-06-26 delta (vision handoff — ADDITIVE to `wire@0.12.0`, NO version bump):** adds a new +> `ImageChunk` variant to the `Chunk` union (`{ type: "image", url, mimeType? }` — `url` is a base64 data +> URL or an `http(s)://` URL) and a transport-facing `ImageInput` (`{ url, mimeType? }`, what a client +> sends on `ChatRequest.images`; the orchestrator converts each into an `ImageChunk` on the persisted user +> message). Vision-capable models receive image chunks natively; non-vision models never see them directly +> — the orchestrator's vision handoff transcribes each to a text description (persisted as a separate +> `text` chunk in the SAME user message). See `backend-handoff.md` §2j. +> > **2026-06-23 delta (workspaces handoff — package bumped `0.11.0` → `0.12.0`, ADDITIVE):** adds > `Workspace` + `WorkspaceEntry` (a list entry with a conversation count) and a required > `workspaceId: string` on `ConversationMeta` (`"default"` for legacy/unspecified conversations). A @@ -68,7 +76,8 @@ export type Chunk = | ToolCallChunk | ToolResultChunk | ErrorChunk - | SystemChunk; + | SystemChunk + | ImageChunk; /** A piece of plain text content from the assistant or user. */ export interface TextChunk { @@ -144,6 +153,46 @@ export interface SystemChunk { readonly text: string; } +/** + * An image attached to a message (e.g. a user-pasted screenshot or pasted + * photo). Carries a `url` that is EITHER a base64 data URL + * (`data:image/png;base64,…`) OR an `http(s)://` URL. Vision-capable models + * receive it natively (the provider serializes it to its image-content + * format); non-vision models never see it directly — the orchestrator's + * **vision handoff** transcribes it to a text description (via a + * vision-capable model) and feeds that text instead, so a text-only model can + * still reason about the image's contents. + * + * When a transcription was performed, it is persisted as a separate `text` + * chunk alongside the `image` chunk in the SAME user message, so the + * description is reused on every later turn (no re-transcription) and a + * client renders both the original image and its textual analysis. + */ +export interface ImageChunk { + readonly type: "image"; + /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */ + readonly url: string; + /** + * Optional MIME type of the image (e.g. `"image/png"`). Inferred from the + * data URL when absent; present so a client can render an icon/label without + * parsing the URL. Optional — callers that only have a URL omit it. + */ + readonly mimeType?: string; +} + +/** + * An image a client attaches to a chat message (`ChatRequest.images`). The + * transport-facing input shape; the orchestrator converts each `ImageInput` + * into an `ImageChunk` on the persisted user message. Carries the same `url` + * semantics as `ImageChunk.url`. + */ +export interface ImageInput { + /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */ + readonly url: string; + /** Optional MIME type (e.g. `"image/png"`). Optional — inferred from the data URL when absent. */ + readonly mimeType?: string; +} + /** * A chat message: a role plus an ordered sequence of chunks. Messages are the * unit passed to and from the provider; chunks are the unit persisted and diff --git a/GLOSSARY.md b/GLOSSARY.md index f0dcdd4..03fd848 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -26,6 +26,9 @@ | **steering** | A user message injected into an in-flight turn at the tool-result boundary (drawn from the **message queue**): the model sees it alongside the tool results and may adjust course. Emitted on the chat stream as a `steering` `AgentEvent` (`TurnSteeringEvent`); the queue surface clears on drain (move, don't duplicate). If the turn ends with a non-empty queue (no tool call fired), the queue carries into a NEW turn as its opening prompt (no `steering` event). `wire@0.8.0`. | mid-turn injection, course correction, interruption | | **computer** | A remote SSH target discovered from the system's `~/.ssh/config` — a read-only VIEW, NOT an editable entity (no CRUD store; to add one the user edits `~/.ssh/config`). Backend-canonical (`wire@0.12.0`, additive). On the wire as `Computer` (`{ alias, hostName, port, user, identityFile, knownHost }`) + `ComputerEntry extends Computer` (adds `usageCount`, for `GET /computers`). `alias` IS the **computerId** — the string persisted per-conversation / per-workspace (the computer analog of `cwd`). Resolution is SERVER-owned (never re-implement): per-conversation `computerId` → `workspace.defaultComputerId` → `null`/local. USER-facing only: a tool-execution target forwarded to tools, NEVER part of the model prompt (does not affect prompt caching); the agent never sees it. HTTP API (`GET /computers`, `GET`/`PUT /conversations/:id/computer`, `PUT /workspaces/:id/default-computer`, `GET /computers/:alias/status`, `POST /computers/:alias/test`) consumed in handoff #2. | ssh host, remote host, server, connection target | | **computerId** | The string id of a **computer** — an SSH config `Host` alias users select. Persisted per-conversation and per-workspace (the computer analog of `cwd`/`workspaceId`). `null` means local (no SSH; today's behavior). On `Workspace` as the REQUIRED `defaultComputerId: string \| null` (null = local / no SSH; the computer analog of `defaultCwd`); per-conversation persistence via `GET`/`PUT`/`DELETE /conversations/:id/computer`. `chat.send` need not send it (resolved server-side from the persisted per-conversation value in the MVP). | ssh alias, host id, remote id | +| **image chunk** | An `ImageChunk` (`{ type: "image", url, mimeType? }`) — a NEW `Chunk` variant for an image attached to a message (a user-pasted screenshot/photo). `url` is a base64 data URL (`data:image/…;base64,…`) OR an `http(s)://` URL. Backend-canonical (`wire@0.12.0`, additive). A user message may be multi-chunk (`[text, image, image, …]` in order). On the wire as `ImageChunk` (persisted) + `ImageInput` (what `ChatRequest.images` carries; the orchestrator converts each into an `ImageChunk`). | picture, photo attachment, screenshot chunk | +| **vision** (capability) | Whether a model can natively accept image input (multimodal). On the wire as `ModelMetadata.vision?: boolean` (`GET /models` `modelInfo[name].vision`). `true` (e.g. any `kimi/*` model) → image chunks are passed through to the provider natively. Absent/`false` (e.g. `umans/glm-5.2`) → the server's **vision handoff** transcribes each image to a text description before the model sees it. The FE shows a vision badge in the model picker; it does NOT decide handoff (server-owned). | multimodal, image support | +| **vision handoff** | The server-owned mechanism by which a NON-vision model still reasons about a pasted image: the orchestrator transcribes the image to a text description (via a vision-capable model) and feeds that text instead, so a text-only model never sees the image directly. The persisted user message keeps the original `image` chunk AND a `text` transcription chunk (`[Image analysis (via )]: …`) in the SAME message — the description is reused on every later turn (no re-transcription). The FE renders both; it sends the image and lets the server decide. Distinct from a vision-capable model, which receives the image natively. | image transcription, vision relay | ## Frontend-specific | Term | Meaning | Aliases to avoid | diff --git a/backend-handoff.md b/backend-handoff.md index aa61fc1..741eb25 100644 --- a/backend-handoff.md +++ b/backend-handoff.md @@ -5,10 +5,11 @@ > **From:** dispatch-web orchestrator · **To:** `../backend` orchestrator · **Courier:** the user. > `lsp` does NOT span the repos (AGENTS.md § Backend seam) — every cross-repo ask flows through here. -_Last updated: 2026-06-26 (§2i ADDED — Heartbeat next-run countdown timer: FE shows a live "Next run in Xm Ys" countdown -from a 1s clock; opens 1 backend ask CR-HB-3: new `GET /workspaces/:id/heartbeat/next-run` → `{ nextRunAt: ISO|null }`. -FE falls back to an approximation (latest run + interval) until the endpoint ships. typecheck 0/0, 865 tests green, biome -clean, build OK. §2h/§2g/§2f unchanged.)_ +_Last updated: 2026-06-26 (§2j ADDED — Vision & vision handoff: image paste in the composer, `image` chunks +rendered in the transcript, and a vision badge in the model picker. Additive to `wire@0.12.0` / +`transport-contract@0.22.0` (NO version bump): new `ImageChunk` in the `Chunk` union + `ImageInput`; +`ChatRequest.images` (`ChatSendMessage` carries it; `ChatQueueMessage` does NOT — steering is text-only); +`ModelMetadata.vision`. typecheck 0/0, 901 tests green (+34), biome clean, build OK. §2i unchanged.)_ **FE is current on `ui-contract@0.2.0` / `transport-contract@0.22.0` / `wire@0.12.0`.** Open asks: **CR-9** (`system:os` should detect WSL + include Linux distro — backend behavior change, no contract bump). The SSH-divergence (§2d) is RESOLVED. @@ -716,6 +717,86 @@ down, confirm it matches when a run actually fires). Until CR-HB-3 ships, the FE --- +## 2j. Vision & vision handoff → **CONSUMED ✅ (backend shipped; FE built + verified)** + +The backend shipped image/vision support: a user can attach images to a chat message, vision-capable +models receive them natively, and non-vision models get an auto-transcribed text description (the +"vision handoff"). The FE now pastes/picks/drops images in the composer, renders `image` chunks in the +transcript, and shows a vision badge in the model picker. Additive to `wire@0.12.0` / +`transport-contract@0.22.0` (**NO version bump** — `ImageChunk`/`ImageInput` were added to the existing +versions; the FE's `file:` dep picks them up automatically, no re-pin needed). + +**New wire/transport types consumed + re-mirrored:** +- `ImageChunk` (`{ type: "image", url, mimeType? }`) — a NEW `Chunk` variant. `url` is a base64 data URL + (`data:image/…;base64,…`) OR an `http(s)://` URL. `ImageInput` (`{ url, mimeType? }`) is the transport- + facing input shape (`ChatRequest.images`); the orchestrator converts each into an `ImageChunk` on the + persisted user message. +- `ChatRequest.images?: readonly ImageInput[]` (so `ChatSendMessage` — which `extends ChatRequest` — + carries it on `chat.send`). `ChatQueueMessage` (steering) does NOT carry `images` — steering is + text-only (correctly: a mid-turn injection has no image surface). +- `ModelMetadata.vision?: boolean` — `true` when the model natively accepts images; absent/`false` → the + server's vision handoff transcribes images to text before the model sees them. +- Re-mirrored `.dispatch/wire.reference.md` (added `ImageChunk` to the `Chunk` union + the `ImageChunk`/ + `ImageInput` interfaces) and `.dispatch/transport-contract.reference.md` (added `images` to `ChatRequest`, + `vision` to `ModelMetadata`, + `ImageChunk`/`ImageInput`/`Computer`/`ComputerEntry` to the re-export). + +**FE (DONE + verified):** +- **Core (`core/chunks`):** the `assertChunkExhaustive` conformance guard caught the new `image` variant + (its purpose) → added the `case "image"` (this was the build break). `appendUserMessage(state, text, + images?)` now echoes a `[text, image, image, …]` user run (text first, then images in order; images-only + when text is empty). The `user-message` event carries ONLY text (never images — images arrive via history/ + loadSince + the optimistic echo), so its de-dup was generalized: it now scans the trailing provisional + USER run for a matching text chunk (not just the last chunk — which would be an image when images were + pasted, causing a duplicate text bubble). `applyHistory`'s during-generation de-dup was generalized to + match a multi-chunk user echo against the trailing committed user run by content equality + (`chunkContentEquals` — text/thinking/image/error/system/tool-call/tool-result), dropping the whole echo + only when fully backed (a partial match is kept until turn-seal drops all provisional wholesale). New + pure helpers `chunkContentEquals` + `trailingRun` (both internal). +16 reducer tests. +- **Transcript (`ChatView.svelte`):** a user `image` chunk renders as an `` (lazy + async-decoded, + max-h-80) inside the user bubble, using the chunk's `url` directly. A non-vision model's persisted user + message keeps the original `image` chunk AND a `text` transcription (`[Image analysis (via )]: …`) + in the SAME message — both render (image, then analysis text). The `read_image` tool call/result + renders like any other tool (its `toolName` is generic — no special-casing). +3 ChatView tests. +- **Composer (`Composer.svelte`):** image paste (clipboard `paste` — extracts image `File` items, + `preventDefault` only when an image is present so text paste still works), an attach-image button + + hidden ``, and drag-drop onto the form. Files are read to + base64 data URLs (`FileReader.readAsDataURL`), capped at 8 MiB, staged as thumbnail previews with + remove buttons. `onSend` signature widened to `(text, images?)`; an image-only send (empty text) is + allowed; `images` is OMITTED on the wire (not `[]`) when none are staged (backward compatible). Steering + (`onQueue`) never forwards images. +6 Composer tests. +- **Store wiring:** `ChatStore.send(text, images?)` forwards `images` on the `chat.send` WS op + echoes + them; `AppStore.send(text, images?)` threads images through the draft→tab promotion; `App.svelte`'s + `handleSend(text, images?)` passes them through. The model catalog already captured `modelInfo` (now + with `vision`); `GET /models` is unchanged. +5 store/app tests. +- **Model picker (`ModelSelector.svelte` + `model-select.ts`):** new pure `isVisionModel(modelInfo, + fullName)`; the model dropdown marks vision-capable models (`" · vision"`), and an indicator below shows + "Vision — this model sees images natively" vs the handoff hint "Pasted images are auto-described". Wired + `modelInfo={store.modelInfo}` from `App.svelte`. +8 model-select/ModelSelector tests. + +**Invariants held:** +- `chat.send` STILL omits `cwd` (the persisted cwd wins) — only `images` was added to the message. +- The `user-message` event still carries only text; images are NEVER expected on it (a watcher fetches + them from history). The de-dup was made robust to the multi-chunk echo rather than reaching for images + on the event. +- `providerRetry`/`generating` unchanged; `image` is a normal committed/provisional chunk (it IS in the + `Chunk.type` union, unlike the transient `provider-retry`), so it persists + replays on reload. +- The `read_image` tool is rendered generically (no surface-id special-casing — the tool-name dispatch is + already identity-free). + +**Verification:** `svelte-check` 0/0; vitest **901/901** (run TWICE — no cross-test pollution; +34 new: +16 reducer, 3 ChatView, 6 Composer, 5 store/app, 4 model-select), biome clean, `vite build` succeeds (the +one CSS warning is PRE-EXISTING — `[file:path]`/`[heartbeat:elapsed]` attribute selectors, unrelated). +**Live probe NOT run:** the backend was not reachable headless at verify time (it is the user's process; +never booted headless). The full data path (paste → data URL → `chat.send` `images` → reducer echo → +transcript render; `GET /models` `vision` → badge; history `image` chunk → render) is covered by unit + +component + store tests. To confirm end-to-end: start the backend, paste an image into the composer with +a vision model selected (e.g. any `kimi/*`), send, confirm the image renders + the model responds to it; +then switch to a non-vision model (e.g. `umans/glm-5.2`), paste an image, send, confirm the image renders +AND a `[Image analysis (via …)]` text bubble appears (the handoff transcription); confirm the vision +badge shows/hides per model in the picker. + +--- + ## 3. Likely NEXT backend asks (heads-up, not yet requested) - **Model max context-window LIMIT** → **CONSUMED ✅** — `GET /models` now returns diff --git a/src/app/App.svelte b/src/app/App.svelte index 09be947..f240a06 100644 --- a/src/app/App.svelte +++ b/src/app/App.svelte @@ -1,5 +1,5 @@
@@ -44,7 +61,32 @@ aria-label="Model selector" > {#each keyModels as model (model)} - + {/each} + {#if selectedVision} +
+ + Vision — this model sees images natively +
+ {:else} +
+ Pasted images are auto-described (vision handoff) +
+ {/if}
-- cgit v1.2.3