From fa0bd9c0e433b1abddc814b48a358c94954c7d36 Mon Sep 17 00:00:00 2001 From: Adam Malczewski Date: Sat, 27 Jun 2026 19:15:04 +0900 Subject: feat(vision): consult_vision tool + vision settings API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backend vision update (additive to wire@0.12.0 / transport-contract@0.22.0, no version bump). Contracts mirrored (.dispatch/transport-contract.reference.md): - VisionSettingsResponse + SetVisionSettingsRequest (GET/PUT /settings/vision). - Delta note: read_image -> consult_vision; numbered placeholders; compaction. Tool rendering (ChatView): - read_image test -> consult_vision (rendering is generic by toolName). - +2 tests: numbered-placeholder text chunk + [Compacted image] text chunk (both regular text chunks, render as-is — no special handling). New vision feature library (src/features/vision/): - logic/view-model.ts (32 tests): VisionSettings/VisionSettingsPatch types (consumer-defines-port), LoadVisionSettings/SaveVisionSettings ports + results, normalizeVisionSettings (network-seam coercion), parseImageLimit/ imageLimitChanged, compactionModelOptions (vision-capable models via chat's public isVisionModel + Auto sentinel), round-trip helpers, imageLimitLabel. - ui/VisionSettingsView.svelte (9 tests): imageLimit input + Save, compactionModel dropdown (Auto + vision-capable models), load-on-mount, save-on-change, error/saved feedback. - index.ts. Cross-unit seam: isVisionModel added to features/chat public index.ts (additive); imported through the public surface, not internals. Store wiring (src/app/store.svelte.ts): - visionSettings state + refreshVisionSettings (GET /settings/vision, normalized at the seam) + setVisionSettings (PUT, partial, returns merged) + VisionSettingsResult; seeded on boot; exposed on AppStore. +4 store tests. Mounted in App.svelte: new "Vision" sidebar view kind + VisionSettingsView in viewContent (not conversation-scoped); load/save adapters; visionManifest. Verification: svelte-check 0/0; vitest 948/948 (run twice, +47 since the prior vision commit); biome clean; vite build OK. See backend-handoff.md §2j. Not merged or pushed. --- .dispatch/transport-contract.reference.md | 37 ++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) (limited to '.dispatch') diff --git a/.dispatch/transport-contract.reference.md b/.dispatch/transport-contract.reference.md index 152a1b0..047f050 100644 --- a/.dispatch/transport-contract.reference.md +++ b/.dispatch/transport-contract.reference.md @@ -13,9 +13,19 @@ > (each entry: `{ url, mimeType? }` — a base64 data URL or `http(s)://` URL; validated non-array/no-url/ > empty-url → 400, empty array treated as absent). `ModelMetadata` gains `vision?: boolean` (true when the > model natively accepts images; absent → the server's vision handoff transcribes images to text before the -> model sees them). `ImageChunk`/`ImageInput` are `@dispatch/wire` types (re-exported here). A non-vision -> model's persisted user message keeps the original `image` chunk AND adds a `text` transcription chunk -> (`[Image analysis (via )]: …`) in the SAME message — render both. See `backend-handoff.md` §2j. +> model sees them). `ImageChunk`/`ImageInput` are `@dispatch/wire` types (re-exported here). +> +> **2026-06-26 update (consult_vision + vision settings — ADDITIVE, NO version bump):** the `read_image` +> tool is REPLACED by `consult_vision` (`{ question: string, imageIds?: number[], path?: string }`) — it +> opens a NEW conversation tab with a vision-capable model, attaches the image + question, and returns the +> vision model's answer (rendered like any tool call/result). Non-vision models now get NUMBERED +> PLACEHOLDERS (`[Image N attached — call consult_vision with imageIds=[N] and a specific question to +> analyze it]`) instead of auto-transcriptions — these are regular `text` chunks (render as-is). Image +> compaction transcribes the oldest images past `imageLimit` to `[Compacted image]: ` text +> chunks (also regular `text` — render as-is; the persisted `image` chunk stays for rendering). NEW global +> vision settings API: `GET /settings/vision` → `VisionSettingsResponse` (`{ imageLimit, compactionModel }`), +> `PUT /settings/vision` ← `SetVisionSettingsRequest` (partial: `imageLimit?` non-negative int, 0 = disable +> compaction; `compactionModel?` `/` or null = auto). See `backend-handoff.md` §2j. > > **2026-06-25 delta (SSH handoff #2 — ADDITIVE to `transport-contract@0.22.0`, NO version bump):** adds the > computer HTTP API types: `ComputerListResponse` (`GET /computers`), `ComputerResponse` (`GET /computers/:alias`), @@ -459,6 +469,27 @@ export interface SystemPromptVariablesResponse { readonly variables: readonly SystemPromptVariable[]; } +// ─── Vision settings (global) ─────────────────────────────────────────────── + +/** + * Response of `GET /settings/vision` — the global vision configuration shared + * across all conversations and vision models. + */ +export interface VisionSettingsResponse { + /** Max native images per turn (default 10); 0 disables image compaction. */ + readonly imageLimit: number; + /** Which model transcribes old images (null = auto-select a vision model). */ + readonly compactionModel: string | null; +} + +/** Body of `PUT /settings/vision` — a partial update. */ +export interface SetVisionSettingsRequest { + /** Non-negative integer (0 = disable compaction). */ + readonly imageLimit?: number; + /** A model name (`/`) or null (auto). */ + readonly compactionModel?: string | null; +} + // ─── Message queue (steering) ───────────────────────────────────────────────── /** -- cgit v1.2.3