summaryrefslogtreecommitdiffhomepage
path: root/.dispatch
diff options
context:
space:
mode:
authorAdam Malczewski <[email protected]>2026-06-27 04:18:59 +0900
committerAdam Malczewski <[email protected]>2026-06-27 04:18:59 +0900
commit3566a20ebbded754070fce66af48690d1a904879 (patch)
treead4941abdbe685e2d4ae85d9ad3d0b6e9cf82c2d /.dispatch
parent2fa03f8d7410c2b8d6be8e10ad088863e83d7177 (diff)
downloaddispatch-web-3566a20ebbded754070fce66af48690d1a904879.tar.gz
dispatch-web-3566a20ebbded754070fce66af48690d1a904879.zip
feat(vision): image paste + transcript image rendering + vision badge
Vision & vision-handoff frontend (consumes the backend's additive [email protected] / [email protected] image types — no version bump). Contracts mirrored: - .dispatch/wire.reference.md: ImageChunk added to the Chunk union + ImageChunk/ImageInput interfaces. - .dispatch/transport-contract.reference.md: ChatRequest.images, ModelMetadata.vision, + ImageChunk/ImageInput re-exports. Core (core/chunks): - conformance: assertChunkExhaustive handles the new 'image' variant (the guard caught it — its purpose). - appendUserMessage(state, text, images?) echoes a [text, image, ...] user run; the user-message event dedup scans the trailing user run (not just the last chunk) so an image-bearing echo doesn't duplicate the text; applyHistory's during-gen dedup matches a multi-chunk echo by content equality (chunkContentEquals + trailingRun helpers). UI: - ChatView renders user 'image' chunks as lazy <img> bubbles; a non-vision model's persisted [image, analysis-text] both render. read_image tool renders generically (no special-casing). - Composer: clipboard paste / file picker / drag-drop of images -> base64 data URLs, thumbnail previews with remove, forwarded on chat.send (omitted when none). Image-only sends allowed; steering (chat.queue) never forwards images. - ModelSelector: vision badge (isVisionModel) marks vision-capable models; indicator shows native-vision vs vision-handoff hint. Store wiring: ChatStore.send + AppStore.send + App.svelte handleSend thread images through; chat.send still omits cwd (only images added). Verification: svelte-check 0/0; vitest 901/901 (run twice, +34 new); biome clean; vite build OK. See backend-handoff.md §2j. Not merged or pushed.
Diffstat (limited to '.dispatch')
-rw-r--r--.dispatch/transport-contract.reference.md39
-rw-r--r--.dispatch/wire.reference.md51
2 files changed, 89 insertions, 1 deletions
diff --git a/.dispatch/transport-contract.reference.md b/.dispatch/transport-contract.reference.md
index 48ba45d..152a1b0 100644
--- a/.dispatch/transport-contract.reference.md
+++ b/.dispatch/transport-contract.reference.md
@@ -8,6 +8,15 @@
> **Orchestrator:** SNAPSHOT of `[email protected]` (MCP status + computers). Regenerate whenever
> it changes.
>
+> **2026-06-26 delta (vision handoff — ADDITIVE, NO version bump):** adds the vision/image surface.
+> `ChatRequest` (+ `ChatSendMessage`/`QueueRequest`) gains an optional `images?: readonly ImageInput[]`
+> (each entry: `{ url, mimeType? }` — a base64 data URL or `http(s)://` URL; validated non-array/no-url/
+> empty-url → 400, empty array treated as absent). `ModelMetadata` gains `vision?: boolean` (true when the
+> model natively accepts images; absent → the server's vision handoff transcribes images to text before the
+> model sees them). `ImageChunk`/`ImageInput` are `@dispatch/wire` types (re-exported here). A non-vision
+> model's persisted user message keeps the original `image` chunk AND adds a `text` transcription chunk
+> (`[Image analysis (via <model>)]: …`) in the SAME message — render both. See `backend-handoff.md` §2j.
+>
> **2026-06-25 delta (SSH handoff #2 — ADDITIVE to `[email protected]`, NO version bump):** adds the
> computer HTTP API types: `ComputerListResponse` (`GET /computers`), `ComputerResponse` (`GET /computers/:alias`),
> `ComputerStatusResponse` (`GET /computers/:alias/status`), `TestComputerResponse` (`POST /computers/:alias/test`),
@@ -59,8 +68,11 @@
import type { SurfaceClientMessage, SurfaceServerMessage } from "@dispatch/ui-contract";
import type {
AgentEvent,
+ Computer,
+ ComputerEntry,
ConversationMeta,
ConversationStatus,
+ ImageInput,
QueuedMessage,
ReasoningEffort,
StoredChunk,
@@ -72,8 +84,12 @@ import type {
export type {
AgentEvent,
CompactionResult,
+ Computer,
+ ComputerEntry,
ConversationMeta,
ConversationStatus,
+ ImageChunk,
+ ImageInput,
QueuedMessage,
ReasoningEffort,
StepMetrics,
@@ -101,6 +117,21 @@ export interface ChatRequest {
readonly message: string;
/**
+ * Images attached to this turn (e.g. a user-pasted screenshot). Each entry's
+ * `url` is a base64 data URL (`data:image/…;base64,…`) or an `http(s)://`
+ * URL. The server converts these to `image` chunks on the persisted user
+ * message. For a VISION-capable model (e.g. kimi), the images are passed
+ * through to the provider natively. For a NON-vision model (e.g. glm-5.2),
+ * the server's vision handoff transcribes each image to a text description
+ * (via a vision-capable model) and feeds that text instead — so a text-only
+ * model can still reason about the image's contents. Optional — omit for a
+ * text-only turn (backward compatible). Validation: non-array `images` →
+ * 400; an image without `url` → 400; empty `url` → 400. An empty array is
+ * accepted and treated as absent.
+ */
+ readonly images?: readonly ImageInput[];
+
+ /**
* The model to use, as a model name in `<credentialName>/<model>` form — one
* of the exact strings returned by `GET /models`. Omit to use the server's
* default credential + model.
@@ -157,6 +188,14 @@ export interface ModelsResponse {
/** Per-model metadata returned alongside the model catalog. */
export interface ModelMetadata {
readonly contextWindow?: number;
+ /**
+ * Whether this model can natively accept image input (vision/multimodal).
+ * When `true`, image chunks in a user message are passed through to the
+ * provider. When `false`/absent, the server's vision handoff transcribes
+ * images to text before the model sees them. A client may use this to show
+ * a vision badge in the model picker. Optional — absent when unknown.
+ */
+ readonly vision?: boolean;
}
/**
diff --git a/.dispatch/wire.reference.md b/.dispatch/wire.reference.md
index 05ed40b..c40fe81 100644
--- a/.dispatch/wire.reference.md
+++ b/.dispatch/wire.reference.md
@@ -6,6 +6,14 @@
>
> **Orchestrator:** SNAPSHOT of `[email protected]` (workspaces + computers). Regenerate whenever `@dispatch/wire` changes.
>
+> **2026-06-26 delta (vision handoff — ADDITIVE to `[email protected]`, NO version bump):** adds a new
+> `ImageChunk` variant to the `Chunk` union (`{ type: "image", url, mimeType? }` — `url` is a base64 data
+> URL or an `http(s)://` URL) and a transport-facing `ImageInput` (`{ url, mimeType? }`, what a client
+> sends on `ChatRequest.images`; the orchestrator converts each into an `ImageChunk` on the persisted user
+> message). Vision-capable models receive image chunks natively; non-vision models never see them directly
+> — the orchestrator's vision handoff transcribes each to a text description (persisted as a separate
+> `text` chunk in the SAME user message). See `backend-handoff.md` §2j.
+>
> **2026-06-23 delta (workspaces handoff — package bumped `0.11.0` → `0.12.0`, ADDITIVE):** adds
> `Workspace` + `WorkspaceEntry` (a list entry with a conversation count) and a required
> `workspaceId: string` on `ConversationMeta` (`"default"` for legacy/unspecified conversations). A
@@ -68,7 +76,8 @@ export type Chunk =
| ToolCallChunk
| ToolResultChunk
| ErrorChunk
- | SystemChunk;
+ | SystemChunk
+ | ImageChunk;
/** A piece of plain text content from the assistant or user. */
export interface TextChunk {
@@ -145,6 +154,46 @@ export interface SystemChunk {
}
/**
+ * An image attached to a message (e.g. a user-pasted screenshot or pasted
+ * photo). Carries a `url` that is EITHER a base64 data URL
+ * (`data:image/png;base64,…`) OR an `http(s)://` URL. Vision-capable models
+ * receive it natively (the provider serializes it to its image-content
+ * format); non-vision models never see it directly — the orchestrator's
+ * **vision handoff** transcribes it to a text description (via a
+ * vision-capable model) and feeds that text instead, so a text-only model can
+ * still reason about the image's contents.
+ *
+ * When a transcription was performed, it is persisted as a separate `text`
+ * chunk alongside the `image` chunk in the SAME user message, so the
+ * description is reused on every later turn (no re-transcription) and a
+ * client renders both the original image and its textual analysis.
+ */
+export interface ImageChunk {
+ readonly type: "image";
+ /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */
+ readonly url: string;
+ /**
+ * Optional MIME type of the image (e.g. `"image/png"`). Inferred from the
+ * data URL when absent; present so a client can render an icon/label without
+ * parsing the URL. Optional — callers that only have a URL omit it.
+ */
+ readonly mimeType?: string;
+}
+
+/**
+ * An image a client attaches to a chat message (`ChatRequest.images`). The
+ * transport-facing input shape; the orchestrator converts each `ImageInput`
+ * into an `ImageChunk` on the persisted user message. Carries the same `url`
+ * semantics as `ImageChunk.url`.
+ */
+export interface ImageInput {
+ /** Image source: a base64 data URL (`data:image/…;base64,…`) or an `http(s)://` URL. */
+ readonly url: string;
+ /** Optional MIME type (e.g. `"image/png"`). Optional — inferred from the data URL when absent. */
+ readonly mimeType?: string;
+}
+
+/**
* A chat message: a role plus an ordered sequence of chunks. Messages are the
* unit passed to and from the provider; chunks are the unit persisted and
* rendered.