diff options
| author | Adam Malczewski <[email protected]> | 2026-06-27 19:15:04 +0900 |
|---|---|---|
| committer | Adam Malczewski <[email protected]> | 2026-06-27 19:15:04 +0900 |
| commit | fa0bd9c0e433b1abddc814b48a358c94954c7d36 (patch) | |
| tree | dedf7adf827de6908b5c7cbb4b4153aba38c1882 | |
| parent | 3566a20ebbded754070fce66af48690d1a904879 (diff) | |
| download | dispatch-web-fa0bd9c0e433b1abddc814b48a358c94954c7d36.tar.gz dispatch-web-fa0bd9c0e433b1abddc814b48a358c94954c7d36.zip | |
feat(vision): consult_vision tool + vision settings API
Backend vision update (additive to [email protected] / [email protected],
no version bump).
Contracts mirrored (.dispatch/transport-contract.reference.md):
- VisionSettingsResponse + SetVisionSettingsRequest (GET/PUT /settings/vision).
- Delta note: read_image -> consult_vision; numbered placeholders; compaction.
Tool rendering (ChatView):
- read_image test -> consult_vision (rendering is generic by toolName).
- +2 tests: numbered-placeholder text chunk + [Compacted image] text chunk
(both regular text chunks, render as-is — no special handling).
New vision feature library (src/features/vision/):
- logic/view-model.ts (32 tests): VisionSettings/VisionSettingsPatch types
(consumer-defines-port), LoadVisionSettings/SaveVisionSettings ports +
results, normalizeVisionSettings (network-seam coercion), parseImageLimit/
imageLimitChanged, compactionModelOptions (vision-capable models via chat's
public isVisionModel + Auto sentinel), round-trip helpers, imageLimitLabel.
- ui/VisionSettingsView.svelte (9 tests): imageLimit input + Save,
compactionModel dropdown (Auto + vision-capable models), load-on-mount,
save-on-change, error/saved feedback.
- index.ts.
Cross-unit seam: isVisionModel added to features/chat public index.ts
(additive); imported through the public surface, not internals.
Store wiring (src/app/store.svelte.ts):
- visionSettings state + refreshVisionSettings (GET /settings/vision,
normalized at the seam) + setVisionSettings (PUT, partial, returns merged) +
VisionSettingsResult; seeded on boot; exposed on AppStore. +4 store tests.
Mounted in App.svelte: new "Vision" sidebar view kind + VisionSettingsView
in viewContent (not conversation-scoped); load/save adapters; visionManifest.
Verification: svelte-check 0/0; vitest 948/948 (run twice, +47 since the
prior vision commit); biome clean; vite build OK. See backend-handoff.md §2j.
Not merged or pushed.
| -rw-r--r-- | .dispatch/transport-contract.reference.md | 37 | ||||
| -rw-r--r-- | GLOSSARY.md | 5 | ||||
| -rw-r--r-- | backend-handoff.md | 73 | ||||
| -rw-r--r-- | src/app/App.svelte | 38 | ||||
| -rw-r--r-- | src/app/store.svelte.ts | 77 | ||||
| -rw-r--r-- | src/app/store.test.ts | 124 | ||||
| -rw-r--r-- | src/features/chat/index.ts | 1 | ||||
| -rw-r--r-- | src/features/chat/ui.test.ts | 57 | ||||
| -rw-r--r-- | src/features/vision/index.ts | 30 | ||||
| -rw-r--r-- | src/features/vision/logic/view-model.test.ts | 198 | ||||
| -rw-r--r-- | src/features/vision/logic/view-model.ts | 189 | ||||
| -rw-r--r-- | src/features/vision/ui/VisionSettingsView.svelte | 192 | ||||
| -rw-r--r-- | src/features/vision/ui/VisionSettingsView.test.ts | 241 |
13 files changed, 1242 insertions, 20 deletions
diff --git a/.dispatch/transport-contract.reference.md b/.dispatch/transport-contract.reference.md index 152a1b0..047f050 100644 --- a/.dispatch/transport-contract.reference.md +++ b/.dispatch/transport-contract.reference.md @@ -13,9 +13,19 @@ > (each entry: `{ url, mimeType? }` — a base64 data URL or `http(s)://` URL; validated non-array/no-url/ > empty-url → 400, empty array treated as absent). `ModelMetadata` gains `vision?: boolean` (true when the > model natively accepts images; absent → the server's vision handoff transcribes images to text before the -> model sees them). `ImageChunk`/`ImageInput` are `@dispatch/wire` types (re-exported here). A non-vision -> model's persisted user message keeps the original `image` chunk AND adds a `text` transcription chunk -> (`[Image analysis (via <model>)]: …`) in the SAME message — render both. See `backend-handoff.md` §2j. +> model sees them). `ImageChunk`/`ImageInput` are `@dispatch/wire` types (re-exported here). +> +> **2026-06-26 update (consult_vision + vision settings — ADDITIVE, NO version bump):** the `read_image` +> tool is REPLACED by `consult_vision` (`{ question: string, imageIds?: number[], path?: string }`) — it +> opens a NEW conversation tab with a vision-capable model, attaches the image + question, and returns the +> vision model's answer (rendered like any tool call/result). Non-vision models now get NUMBERED +> PLACEHOLDERS (`[Image N attached — call consult_vision with imageIds=[N] and a specific question to +> analyze it]`) instead of auto-transcriptions — these are regular `text` chunks (render as-is). Image +> compaction transcribes the oldest images past `imageLimit` to `[Compacted image]: <description>` text +> chunks (also regular `text` — render as-is; the persisted `image` chunk stays for rendering). NEW global +> vision settings API: `GET /settings/vision` → `VisionSettingsResponse` (`{ imageLimit, compactionModel }`), +> `PUT /settings/vision` ← `SetVisionSettingsRequest` (partial: `imageLimit?` non-negative int, 0 = disable +> compaction; `compactionModel?` `<key>/<model>` or null = auto). See `backend-handoff.md` §2j. > > **2026-06-25 delta (SSH handoff #2 — ADDITIVE to `[email protected]`, NO version bump):** adds the > computer HTTP API types: `ComputerListResponse` (`GET /computers`), `ComputerResponse` (`GET /computers/:alias`), @@ -459,6 +469,27 @@ export interface SystemPromptVariablesResponse { readonly variables: readonly SystemPromptVariable[]; } +// ─── Vision settings (global) ─────────────────────────────────────────────── + +/** + * Response of `GET /settings/vision` — the global vision configuration shared + * across all conversations and vision models. + */ +export interface VisionSettingsResponse { + /** Max native images per turn (default 10); 0 disables image compaction. */ + readonly imageLimit: number; + /** Which model transcribes old images (null = auto-select a vision model). */ + readonly compactionModel: string | null; +} + +/** Body of `PUT /settings/vision` — a partial update. */ +export interface SetVisionSettingsRequest { + /** Non-negative integer (0 = disable compaction). */ + readonly imageLimit?: number; + /** A model name (`<key>/<model>`) or null (auto). */ + readonly compactionModel?: string | null; +} + // ─── Message queue (steering) ───────────────────────────────────────────────── /** diff --git a/GLOSSARY.md b/GLOSSARY.md index 03fd848..3973fe1 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -27,8 +27,9 @@ | **computer** | A remote SSH target discovered from the system's `~/.ssh/config` — a read-only VIEW, NOT an editable entity (no CRUD store; to add one the user edits `~/.ssh/config`). Backend-canonical (`[email protected]`, additive). On the wire as `Computer` (`{ alias, hostName, port, user, identityFile, knownHost }`) + `ComputerEntry extends Computer` (adds `usageCount`, for `GET /computers`). `alias` IS the **computerId** — the string persisted per-conversation / per-workspace (the computer analog of `cwd`). Resolution is SERVER-owned (never re-implement): per-conversation `computerId` → `workspace.defaultComputerId` → `null`/local. USER-facing only: a tool-execution target forwarded to tools, NEVER part of the model prompt (does not affect prompt caching); the agent never sees it. HTTP API (`GET /computers`, `GET`/`PUT /conversations/:id/computer`, `PUT /workspaces/:id/default-computer`, `GET /computers/:alias/status`, `POST /computers/:alias/test`) consumed in handoff #2. | ssh host, remote host, server, connection target | | **computerId** | The string id of a **computer** — an SSH config `Host` alias users select. Persisted per-conversation and per-workspace (the computer analog of `cwd`/`workspaceId`). `null` means local (no SSH; today's behavior). On `Workspace` as the REQUIRED `defaultComputerId: string \| null` (null = local / no SSH; the computer analog of `defaultCwd`); per-conversation persistence via `GET`/`PUT`/`DELETE /conversations/:id/computer`. `chat.send` need not send it (resolved server-side from the persisted per-conversation value in the MVP). | ssh alias, host id, remote id | | **image chunk** | An `ImageChunk` (`{ type: "image", url, mimeType? }`) — a NEW `Chunk` variant for an image attached to a message (a user-pasted screenshot/photo). `url` is a base64 data URL (`data:image/…;base64,…`) OR an `http(s)://` URL. Backend-canonical (`[email protected]`, additive). A user message may be multi-chunk (`[text, image, image, …]` in order). On the wire as `ImageChunk` (persisted) + `ImageInput` (what `ChatRequest.images` carries; the orchestrator converts each into an `ImageChunk`). | picture, photo attachment, screenshot chunk | -| **vision** (capability) | Whether a model can natively accept image input (multimodal). On the wire as `ModelMetadata.vision?: boolean` (`GET /models` `modelInfo[name].vision`). `true` (e.g. any `kimi/*` model) → image chunks are passed through to the provider natively. Absent/`false` (e.g. `umans/glm-5.2`) → the server's **vision handoff** transcribes each image to a text description before the model sees it. The FE shows a vision badge in the model picker; it does NOT decide handoff (server-owned). | multimodal, image support | -| **vision handoff** | The server-owned mechanism by which a NON-vision model still reasons about a pasted image: the orchestrator transcribes the image to a text description (via a vision-capable model) and feeds that text instead, so a text-only model never sees the image directly. The persisted user message keeps the original `image` chunk AND a `text` transcription chunk (`[Image analysis (via <model>)]: …`) in the SAME message — the description is reused on every later turn (no re-transcription). The FE renders both; it sends the image and lets the server decide. Distinct from a vision-capable model, which receives the image natively. | image transcription, vision relay | +| **vision** (capability) | Whether a model can natively accept image input (multimodal). On the wire as `ModelMetadata.vision?: boolean` (`GET /models` `modelInfo[name].vision`). `true` (e.g. any `kimi/*` model) → image chunks are passed through to the provider natively. Absent/`false` (e.g. `umans/glm-5.2`) → the server's **vision handoff** gives the model a numbered placeholder + the `consult_vision` tool. The FE shows a vision badge in the model picker; it does NOT decide handoff (server-owned). | multimodal, image support | +| **vision handoff** | The server-owned mechanism by which a NON-vision model still reasons about a pasted image. A non-vision model gets a NUMBERED PLACEHOLDER text chunk alongside the persisted `image` chunk, then calls the `consult_vision` tool (which opens a NEW conversation tab with a vision-capable model, attaches the image + question, and returns the vision model's answer). The persisted user message keeps the original `image` chunk (the FE renders it) AND the placeholder text. When a vision-capable model has more than `imageLimit` images in history, image **compaction** transcribes the oldest to `[Compacted image]: <description>` text chunks (the `image` chunk stays for rendering). All these are regular `text` chunks — the FE renders them as-is. Distinct from a vision-capable model, which receives the image natively. | image transcription, vision relay | +| **consult_vision** | A tool available to all models that defers image analysis to a vision-capable model: it opens a NEW conversation tab (with a vision model, e.g. Kimi), attaches the image (by `imageIds` from a pasted placeholder, or by `path` from disk) + a `question`, and returns the conversation id + the vision model's answer (suggesting the dispatch CLI for follow-ups). Replaces the former `read_image` tool. Rendered generically like any tool call/result (by `toolName`). | read_image (former), vision tool | ## Frontend-specific | Term | Meaning | Aliases to avoid | diff --git a/backend-handoff.md b/backend-handoff.md index 741eb25..9338a5a 100644 --- a/backend-handoff.md +++ b/backend-handoff.md @@ -5,11 +5,12 @@ > **From:** dispatch-web orchestrator · **To:** `../backend` orchestrator · **Courier:** the user. > `lsp` does NOT span the repos (AGENTS.md § Backend seam) — every cross-repo ask flows through here. -_Last updated: 2026-06-26 (§2j ADDED — Vision & vision handoff: image paste in the composer, `image` chunks -rendered in the transcript, and a vision badge in the model picker. Additive to `[email protected]` / -`[email protected]` (NO version bump): new `ImageChunk` in the `Chunk` union + `ImageInput`; -`ChatRequest.images` (`ChatSendMessage` carries it; `ChatQueueMessage` does NOT — steering is text-only); -`ModelMetadata.vision`. typecheck 0/0, 901 tests green (+34), biome clean, build OK. §2i unchanged.)_ +_Last updated: 2026-06-26 (§2j UPDATED — consult_vision tool + vision settings API: `read_image` is REPLACED by +`consult_vision` (non-vision models now get numbered placeholder text chunks, not auto-transcriptions); NEW global +`GET`/`PUT /settings/vision` (`VisionSettingsResponse`/`SetVisionSettingsRequest` — `imageLimit` + `compactionModel`); +image compaction transcribes old images to `[Compacted image]: …` text chunks (all regular text — render as-is). New +`vision` feature library + "Vision" sidebar view (imageLimit input + compactionModel dropdown of vision-capable models + +"Auto"). typecheck 0/0, 948 tests green (+47), biome clean, build OK. §2i unchanged.)_ **FE is current on `[email protected]` / `[email protected]` / `[email protected]`.** Open asks: **CR-9** (`system:os` should detect WSL + include Linux distro — backend behavior change, no contract bump). The SSH-divergence (§2d) is RESOLVED. @@ -792,8 +793,66 @@ transcript render; `GET /models` `vision` → badge; history `image` chunk → r component + store tests. To confirm end-to-end: start the backend, paste an image into the composer with a vision model selected (e.g. any `kimi/*`), send, confirm the image renders + the model responds to it; then switch to a non-vision model (e.g. `umans/glm-5.2`), paste an image, send, confirm the image renders -AND a `[Image analysis (via …)]` text bubble appears (the handoff transcription); confirm the vision -badge shows/hides per model in the picker. +AND a numbered placeholder `[Image N attached — call consult_vision…]` text bubble appears (the non-vision +handoff — see the update below; the old `[Image analysis (via …)]` auto-transcription is GONE); confirm +the vision badge shows/hides per model in the picker. + +### 2j-update. consult_vision tool + vision settings API → **CONSUMED ✅ (backend updated; FE built + verified)** + +A follow-up to §2j. The backend revised the vision surface: the `read_image` tool is REPLACED by +`consult_vision`, non-vision models get NUMBERED PLACEHOLDERS (not auto-transcriptions), and there is a +NEW global vision-settings API. Additive to `[email protected]` / `[email protected]` (NO version bump); +re-mirrored `.dispatch/transport-contract.reference.md` (added `VisionSettingsResponse` / +`SetVisionSettingsRequest` + a delta note). + +**What changed (backend):** +- **`read_image` → `consult_vision`** (`{ question: string (req), imageIds?: number[], path?: string }`): + opens a NEW conversation tab with a vision-capable model (Kimi), attaches the image + question, returns + the conversation id + the vision model's answer (suggests the dispatch CLI for follow-ups). Rendered like + any tool call/result (generic `toolName` dispatch — no special-casing). +- **Non-vision models get NUMBERED PLACEHOLDERS** instead of auto-transcriptions: a pasted image on a + non-vision model persists an `image` chunk + a `text` chunk `[Image N attached — call consult_vision + with imageIds=[N] and a specific question to analyze it]`. (The old `[Image analysis (via <model>)]: …` + auto-transcription is GONE.) +- **Image compaction** (transparent): when a vision model has > `imageLimit` images in history, the oldest + are transcribed to `[Compacted image]: <description>` `text` chunks (the persisted `image` chunk stays + for rendering). Both placeholder + compacted chunks are REGULAR `text` chunks — render as-is (no special + handling). +- **NEW global vision settings API:** `GET /settings/vision` → `VisionSettingsResponse` + (`{ imageLimit: number (default 10), compactionModel: string|null (null = auto) }`); `PUT /settings/vision` + ← `SetVisionSettingsRequest` (partial: `imageLimit?` non-negative int, 0 = disable compaction; + `compactionModel?` `<key>/<model>` or null). + +**FE (DONE + verified):** +- **Tool rendering:** the ChatView `read_image` test → `consult_vision` (rendering is generic — renders by + `toolName`, so no component change; only the test name/input updated). +2 ChatView tests for the + placeholder text chunk + the compacted-image text chunk (both render as regular text). +- **New `vision` feature library** (`src/features/vision/`): pure `logic/view-model.ts` (32 tests) — + `VisionSettings`/`VisionSettingsPatch` (owned locally, consumer-defines-port; the shapes are a plain REST + surface, mirroring heartbeat/mcp), `LoadVisionSettings`/`SaveVisionSettings` ports + result types, + `normalizeVisionSettings` (network-seam coercion — a malformed body can't crash the renderer), + `parseImageLimit`/`imageLimitChanged` (dirty-check), `compactionModelOptions` (filters `GET /models` to + vision-capable models via the chat feature's public `isVisionModel` export + an "Auto" sentinel), + `selectedCompactionValue`/`compactionModelFromValue` round-trip, `imageLimitLabel`; `ui/VisionSettingsView.svelte` + (imageLimit text input + Save, compactionModel dropdown with Auto + vision-capable models, load-on-mount, + save-on-change, error/saved feedback; 9 component tests); `index.ts`. +- **Cross-unit seam:** `isVisionModel` was added to `features/chat`'s public `index.ts` (additive — the + vision feature imports it through the public surface, not the chat internals). +- **Store wiring** (`src/app/store.svelte.ts`): `visionSettings` reactive state + `refreshVisionSettings()` + (`GET /settings/vision`, normalized at the seam) + `setVisionSettings(patch)` (`PUT /settings/vision`, + returns merged settings) + `VisionSettingsResult`; seeded on boot; exposed on `AppStore`. +4 store tests. +- **Mounted in `App.svelte`:** a new "Vision" sidebar view kind (`viewKinds`) + `VisionSettingsView` in the + `viewContent` snippet (not conversation-scoped — no `{#key}`); `loadVisionSettings`/`saveVisionSettings` + adapters wrap the store; `visionManifest` in `loadedModules`. + +**Verification:** `svelte-check` 0/0; vitest **948/948** (run TWICE — no cross-test pollution; +47 new +since the §2j baseline: 32 vision view-model, 9 VisionSettingsView, 4 store, +2 ChatView +placeholder/compacted, and the read_image→consult_vision test update), biome clean, `vite build` succeeds. +**Live probe NOT run** (backend not reachable headless). To confirm end-to-end: open the Vision sidebar +view, confirm the imageLimit + compactionModel load; change imageLimit → Save → confirm it persists on +reload; set compactionModel to a vision model → confirm the dropdown reflects it; paste an image with a +non-vision model → confirm the `[Image N attached — call consult_vision…]` placeholder renders (not an +auto-transcription); trigger a `consult_vision` tool call → confirm it renders like a tool. --- diff --git a/src/app/App.svelte b/src/app/App.svelte index f240a06..f41d7ba 100644 --- a/src/app/App.svelte +++ b/src/app/App.svelte @@ -77,6 +77,13 @@ type SaveSystemPrompt as SaveSystemPromptAlias, manifest as systemPromptManifest, } from "../features/system-prompt"; + import { + VisionSettingsView, + manifest as visionManifest, + type LoadVisionSettingsResult, + type SaveVisionSettingsResult, + type VisionSettingsPatch, + } from "../features/vision"; import type { AppStore } from "./store.svelte"; import ErrorModal from "./ErrorModal.svelte"; import { createLocalStore } from "../adapters/local-storage"; @@ -106,6 +113,7 @@ { id: "cache-warming", label: "Cache Warming" }, { id: "tasks", label: "Tasks" }, { id: "compaction", label: "Compaction" }, + { id: "vision", label: "Vision" }, { id: "heartbeat", label: "Heartbeat" }, { id: "system-prompt", label: "System Prompt" }, { id: "settings", label: "Settings" }, @@ -143,6 +151,7 @@ settingsManifest, systemPromptManifest, heartbeatManifest, + visionManifest, ].map((m) => [m.name, m.description] as const); // Smart-scroll: keep the transcript pinned to the bottom while it streams, @@ -304,6 +313,27 @@ : { ok: false, error: result.error }; } + // Adapt the store's global vision-settings API to the vision feature's ports. + async function loadVisionSettings(): Promise<LoadVisionSettingsResult> { + // The store seeds `visionSettings` on boot; a refresh keeps it current. + await store.refreshVisionSettings(); + const settings = store.visionSettings; + if (settings === null) { + return { ok: false, error: "Vision settings not available." }; + } + return { ok: true, settings }; + } + + async function saveVisionSettings( + patch: VisionSettingsPatch, + ): Promise<SaveVisionSettingsResult> { + const result = await store.setVisionSettings(patch); + if (result === null) return { ok: false, error: "Vision settings not available." }; + return result.ok + ? { ok: true, settings: result.settings } + : { ok: false, error: result.error }; + } + // Adapt the store's chat-limit result to the settings feature's port. On a // raise the active chat refills (prepends older history); preserve the // reader's viewport over the prepend (the manual analogue of CSS scroll @@ -660,6 +690,14 @@ savePercent={saveCompactPercent} /> {/key} + {:else if kind === "vision"} + <!-- Global vision settings (image compaction). Not conversation-scoped (no {#key}). --> + <VisionSettingsView + models={store.models} + modelInfo={store.modelInfo} + load={loadVisionSettings} + save={saveVisionSettings} + /> {:else if kind === "system-prompt"} <!-- Global system prompt template. Opens a full-page modal editor (half template / half variable palette). Not conversation-scoped (no {#key}). --> diff --git a/src/app/store.svelte.ts b/src/app/store.svelte.ts index 41f3a5d..554d5e0 100644 --- a/src/app/store.svelte.ts +++ b/src/app/store.svelte.ts @@ -27,6 +27,7 @@ import type { SetReasoningEffortRequest, SetSystemPromptTemplateRequest, SetTitleRequest, + SetVisionSettingsRequest, SystemPromptTemplateResponse, SystemPromptVariable, SystemPromptVariablesResponse, @@ -67,6 +68,11 @@ import type { import { normalizeHeartbeatConfig, normalizeHeartbeatRuns } from "../features/heartbeat"; import type { Tab, TabsState } from "../features/tabs"; import { createTabsStore, deriveTitle, type TabsStore } from "../features/tabs"; +import { + normalizeVisionSettings, + type VisionSettings, + type VisionSettingsPatch, +} from "../features/vision"; import { resolveHttpUrl } from "./resolve-http-url"; import { resolveWsUrl } from "./resolve-ws-url"; import { randomId } from "./uuid"; @@ -123,6 +129,11 @@ export type CompactPercentResult = | { readonly ok: true; readonly percent: number } | { readonly ok: false; readonly error: string }; +/** Outcome of `PUT /settings/vision` (global vision-settings save). */ +export type VisionSettingsResult = + | { readonly ok: true; readonly settings: VisionSettings } + | { readonly ok: false; readonly error: string }; + /** Outcome of `GET /system-prompt` (global template load). */ export type SystemPromptLoadResult = | { readonly ok: true; readonly template: string } @@ -269,6 +280,23 @@ export interface AppStore { */ setCompactPercent(percent: number): Promise<CompactPercentResult | null>; /** + * The GLOBAL vision settings (`GET /settings/vision`): `imageLimit` (max + * native images per turn before compaction; 0 = disabled) + `compactionModel` + * (which vision model transcribes old images; null = auto). Shared across all + * conversations. Seeded on boot; `null` = not yet fetched. + */ + readonly visionSettings: VisionSettings | null; + /** + * Refetch the global vision settings (`GET /settings/vision`). Called by the + * vision-settings view on mount; also seeded on boot. + */ + refreshVisionSettings(): Promise<void>; + /** + * Save a PARTIAL vision-settings update (`PUT /settings/vision`). Either + * field may be omitted. Returns the merged settings on success. + */ + setVisionSettings(patch: VisionSettingsPatch): Promise<VisionSettingsResult | null>; + /** * Fetch the workspace conversation's language-server status (`GET /conversations/:id/lsp`). * The backend lazily spawns servers, so this may take a moment on the first call for a cwd. */ @@ -626,6 +654,22 @@ export function createAppStore(opts?: CreateAppStoreOptions): AppStore { // backend on focus change; null = not yet fetched. 0 = disabled. let compactPercent = $state<number | null>(null); + // The GLOBAL vision settings (shared across all conversations). Seeded on + // boot; null = not yet fetched. + let visionSettings = $state<VisionSettings | null>(null); + + /** Refetch the global vision settings (`GET /settings/vision`). */ + async function refreshVisionSettings(): Promise<void> { + try { + const res = await fetchImpl(`${httpBase}/settings/vision`); + if (!res.ok) return; + const data = normalizeVisionSettings(await res.json()); + visionSettings = data; + } catch (err) { + reportError("Failed to load vision settings", err); + } + } + /** Refetch the workspace conversation's compact percent (works for a draft too). */ async function refreshCompactPercent(): Promise<void> { const id = workspaceConversationId(); @@ -1066,6 +1110,7 @@ export function createAppStore(opts?: CreateAppStoreOptions): AppStore { void refreshModel(); void refreshReasoningEffort(); void refreshCompactPercent(); + void refreshVisionSettings(); // Fetch the authoritative open-conversation list from the backend (cross- // device tab sync). Merges with the localStorage-restored tabs: opens new @@ -1142,6 +1187,12 @@ export function createAppStore(opts?: CreateAppStoreOptions): AppStore { get compactPercent(): number | null { return compactPercent; }, + get visionSettings(): VisionSettings | null { + return visionSettings; + }, + async refreshVisionSettings(): Promise<void> { + await refreshVisionSettings(); + }, get chatLimit(): number { return chatLimit; }, @@ -1500,6 +1551,32 @@ export function createAppStore(opts?: CreateAppStoreOptions): AppStore { } }, + async setVisionSettings(patch: VisionSettingsPatch): Promise<VisionSettingsResult | null> { + const body: SetVisionSettingsRequest = patch; + try { + const res = await fetchImpl(`${httpBase}/settings/vision`, { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + }); + if (!res.ok) { + const errBody = (await res.json().catch(() => null)) as { error?: string } | null; + return { + ok: false, + error: errBody?.error ?? `Set vision settings failed (HTTP ${res.status})`, + }; + } + const data = normalizeVisionSettings(await res.json()); + visionSettings = data; + return { ok: true, settings: data }; + } catch (err) { + return { + ok: false, + error: err instanceof Error ? err.message : "Set vision settings request failed", + }; + } + }, + async setChatLimit(limit: number): Promise<ChatLimitResult> { const next = normalizeChatLimit(limit); chatLimitStore.save(next); diff --git a/src/app/store.test.ts b/src/app/store.test.ts index 7130c4d..1534402 100644 --- a/src/app/store.test.ts +++ b/src/app/store.test.ts @@ -1404,3 +1404,127 @@ describe("createAppStore", () => { store.dispose(); }); }); + +describe("createAppStore — vision settings (global)", () => { + function visionFetch(initial: { imageLimit: number; compactionModel: string | null }): { + fetchImpl: typeof fetch; + puts: { imageLimit?: number; compactionModel?: string | null }[]; + } { + let current = initial; + const puts: { imageLimit?: number; compactionModel?: string | null }[] = []; + return { + puts, + fetchImpl: async (input: string | URL | Request, init?: RequestInit): Promise<Response> => { + const url = + typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + if (url.endsWith("/models")) { + return new Response( + JSON.stringify({ + models: ["kimi/k2", "umans/glm-5.2"], + modelInfo: { "kimi/k2": { vision: true } }, + }), + { status: 200 }, + ); + } + if (url.endsWith("/settings/vision")) { + if (init?.method === "PUT") { + const text = typeof init.body === "string" ? init.body : ""; + const body = text ? (JSON.parse(text) as object) : {}; + puts.push(body as { imageLimit?: number; compactionModel?: string | null }); + current = { ...current, ...(body as object) } as { + imageLimit: number; + compactionModel: string | null; + }; + } + return new Response(JSON.stringify(current), { status: 200 }); + } + // Default: empty history + no cwd for the other endpoints. + return new Response(JSON.stringify({ chunks: [], latestSeq: 0 }), { status: 200 }); + }, + }; + } + + it("loads vision settings on boot (GET /settings/vision)", async () => { + const { fetchImpl } = visionFetch({ imageLimit: 7, compactionModel: "kimi/k2" }); + const store = createAppStore({ + socketFactory: () => fakeSocket(), + fetchImpl, + localStorage: createFakeStorage(), + }); + fakeSocket().resolveOpen(); // not strictly needed for HTTP + + await vi.waitFor(() => { + expect(store.visionSettings).toEqual({ imageLimit: 7, compactionModel: "kimi/k2" }); + }); + store.dispose(); + }); + + it("setVisionSettings PUTs a partial update and reflects the merged settings", async () => { + const ctx = visionFetch({ imageLimit: 10, compactionModel: null }); + const store = createAppStore({ + socketFactory: () => fakeSocket(), + fetchImpl: ctx.fetchImpl, + localStorage: createFakeStorage(), + }); + + await vi.waitFor(() => { + expect(store.visionSettings?.imageLimit).toBe(10); + }); + + const result = await store.setVisionSettings({ imageLimit: 3 }); + expect(result?.ok).toBe(true); + if (result?.ok) { + expect(result.settings.imageLimit).toBe(3); + expect(result.settings.compactionModel).toBeNull(); + } + expect(ctx.puts).toEqual([{ imageLimit: 3 }]); + expect(store.visionSettings?.imageLimit).toBe(3); + + // A second save updates compactionModel only. + const result2 = await store.setVisionSettings({ compactionModel: "kimi/k2" }); + expect(result2?.ok).toBe(true); + expect(ctx.puts).toEqual([{ imageLimit: 3 }, { compactionModel: "kimi/k2" }]); + expect(store.visionSettings?.compactionModel).toBe("kimi/k2"); + store.dispose(); + }); + + it("refreshVisionSettings refetches (load adapter)", async () => { + const { fetchImpl } = visionFetch({ imageLimit: 5, compactionModel: null }); + const store = createAppStore({ + socketFactory: () => fakeSocket(), + fetchImpl, + localStorage: createFakeStorage(), + }); + + await store.refreshVisionSettings(); + expect(store.visionSettings).toEqual({ imageLimit: 5, compactionModel: null }); + store.dispose(); + }); + + it("surfaces a PUT error", async () => { + const fetchImpl: typeof fetch = async (input, init) => { + const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + if (url.endsWith("/settings/vision")) { + if (init?.method === "PUT") { + return new Response(JSON.stringify({ error: "invalid imageLimit" }), { status: 400 }); + } + return new Response(JSON.stringify({ imageLimit: 10, compactionModel: null }), { + status: 200, + }); + } + return new Response(JSON.stringify({ models: [] }), { status: 200 }); + }; + const store = createAppStore({ + socketFactory: () => fakeSocket(), + fetchImpl, + localStorage: createFakeStorage(), + }); + + const result = await store.setVisionSettings({ imageLimit: -1 }); + expect(result?.ok).toBe(false); + if (result !== null && !result.ok) { + expect(result.error).toContain("invalid imageLimit"); + } + store.dispose(); + }); +}); diff --git a/src/features/chat/index.ts b/src/features/chat/index.ts index ddb094d..8694691 100644 --- a/src/features/chat/index.ts +++ b/src/features/chat/index.ts @@ -6,6 +6,7 @@ export type { } from "../../core/chunks"; export { groupRenderedChunks, viewProviderRetry } from "../../core/chunks"; export type { TurnMetricsEntry } from "../../core/metrics"; +export { isVisionModel } from "./model-select"; export type { ChatTransport, HistorySync, HistoryWindow, MetricsSync } from "./ports"; export type { EffortOption, diff --git a/src/features/chat/ui.test.ts b/src/features/chat/ui.test.ts index af32f72..b0aa6f0 100644 --- a/src/features/chat/ui.test.ts +++ b/src/features/chat/ui.test.ts @@ -655,8 +655,10 @@ describe("ChatView", () => { expect(container.querySelector("img")?.getAttribute("src")).toBe(url); }); - it("renders a read_image tool call/result like any other tool", () => { - // The read_image tool is available to all models; it is a normal tool call. + it("renders a consult_vision tool call/result like any other tool", () => { + // read_image is GONE — replaced by consult_vision (opens a vision-model + // conversation, attaches the image + question, returns the answer). It is + // a normal tool call: rendered generically by toolName. const chunks: RenderedChunk[] = [ { seq: 1, @@ -664,8 +666,8 @@ describe("ChatView", () => { chunk: { type: "tool-call", toolCallId: "tc1", - toolName: "read_image", - input: { path: "/tmp/screenshot.png" }, + toolName: "consult_vision", + input: { question: "what is in this image?", imageIds: [1] }, }, provisional: false, }, @@ -675,8 +677,8 @@ describe("ChatView", () => { chunk: { type: "tool-result", toolCallId: "tc1", - toolName: "read_image", - content: "a screenshot of the desktop", + toolName: "consult_vision", + content: "a red square on a white background", isError: false, }, provisional: false, @@ -685,8 +687,47 @@ describe("ChatView", () => { render(ChatView, { props: { chunks } }); - expect(screen.getAllByText("read_image").length).toBeGreaterThan(0); - expect(screen.getByText("a screenshot of the desktop")).toBeInTheDocument(); + expect(screen.getAllByText("consult_vision").length).toBeGreaterThan(0); + expect(screen.getByText("a red square on a white background")).toBeInTheDocument(); + }); + + it("renders a non-vision placeholder text chunk as-is", () => { + // A non-vision model gets a numbered placeholder (a regular text chunk) + // instead of an auto-transcription. Renders like any text chunk. + const chunks: RenderedChunk[] = [ + { + seq: 1, + role: "user", + chunk: { + type: "text", + text: "[Image 1 attached — call consult_vision with imageIds=[1] and a specific question to analyze it]", + }, + provisional: false, + }, + ]; + + render(ChatView, { props: { chunks } }); + + expect(screen.getByText(/\[Image 1 attached/)).toBeInTheDocument(); + expect(screen.getByText(/consult_vision with imageIds/)).toBeInTheDocument(); + }); + + it("renders a compacted-image text chunk as-is", () => { + // Image compaction transcribes old images to [Compacted image]: <desc>. + // Regular text chunk — render as-is. + const chunks: RenderedChunk[] = [ + { + seq: 1, + role: "user", + chunk: { type: "text", text: "[Compacted image]: a chart showing rising sales" }, + provisional: false, + }, + ]; + + render(ChatView, { props: { chunks } }); + + expect(screen.getByText(/\[Compacted image\]/)).toBeInTheDocument(); + expect(screen.getByText(/rising sales/)).toBeInTheDocument(); }); }); diff --git a/src/features/vision/index.ts b/src/features/vision/index.ts new file mode 100644 index 0000000..6956d75 --- /dev/null +++ b/src/features/vision/index.ts @@ -0,0 +1,30 @@ +export type { + CompactionModelOption, + ImageLimitParse, + LoadVisionSettings, + LoadVisionSettingsResult, + SaveVisionSettings, + SaveVisionSettingsResult, + VisionSettings, + VisionSettingsPatch, +} from "./logic/view-model"; +export { + AUTO_COMPACTION_MODEL, + compactionModelChanged, + compactionModelFromValue, + compactionModelOptions, + DEFAULT_IMAGE_LIMIT, + imageLimitChanged, + imageLimitLabel, + MAX_IMAGE_LIMIT, + normalizeVisionSettings, + parseImageLimit, + selectedCompactionValue, +} from "./logic/view-model"; +export { default as VisionSettingsView } from "./ui/VisionSettingsView.svelte"; + +/** Public module manifest — aggregated by the shell's "Loaded Modules" view. */ +export const manifest = { + name: "vision", + description: "Global vision settings (image compaction limit + model)", +} as const; diff --git a/src/features/vision/logic/view-model.test.ts b/src/features/vision/logic/view-model.test.ts new file mode 100644 index 0000000..b1db822 --- /dev/null +++ b/src/features/vision/logic/view-model.test.ts @@ -0,0 +1,198 @@ +import type { ModelMetadata } from "@dispatch/transport-contract"; +import { describe, expect, it } from "vitest"; +import { + AUTO_COMPACTION_MODEL, + compactionModelChanged, + compactionModelFromValue, + compactionModelOptions, + DEFAULT_IMAGE_LIMIT, + imageLimitChanged, + imageLimitLabel, + MAX_IMAGE_LIMIT, + normalizeVisionSettings, + parseImageLimit, + selectedCompactionValue, +} from "./view-model"; + +describe("normalizeVisionSettings", () => { + it("returns the value as-is for a well-formed body", () => { + expect(normalizeVisionSettings({ imageLimit: 5, compactionModel: "kimi/k2" })).toEqual({ + imageLimit: 5, + compactionModel: "kimi/k2", + }); + }); + + it("coerces a null compactionModel to null", () => { + expect(normalizeVisionSettings({ imageLimit: 10, compactionModel: null })).toEqual({ + imageLimit: 10, + compactionModel: null, + }); + }); + + it("defaults imageLimit when absent or non-numeric", () => { + expect(normalizeVisionSettings({ compactionModel: "kimi/k2" })).toEqual({ + imageLimit: DEFAULT_IMAGE_LIMIT, + compactionModel: "kimi/k2", + }); + expect(normalizeVisionSettings({ imageLimit: "oops", compactionModel: null })).toEqual({ + imageLimit: DEFAULT_IMAGE_LIMIT, + compactionModel: null, + }); + }); + + it("floors a fractional imageLimit", () => { + expect(normalizeVisionSettings({ imageLimit: 7.9, compactionModel: null }).imageLimit).toBe(7); + }); + + it("defaults for a null/non-object body (never crashes)", () => { + expect(normalizeVisionSettings(null)).toEqual({ + imageLimit: DEFAULT_IMAGE_LIMIT, + compactionModel: null, + }); + expect(normalizeVisionSettings("nope")).toEqual({ + imageLimit: DEFAULT_IMAGE_LIMIT, + compactionModel: null, + }); + }); + + it("treats a negative imageLimit as the default", () => { + expect(normalizeVisionSettings({ imageLimit: -3, compactionModel: null }).imageLimit).toBe( + DEFAULT_IMAGE_LIMIT, + ); + }); + + it("treats an empty compactionModel string as null", () => { + expect( + normalizeVisionSettings({ imageLimit: 10, compactionModel: "" }).compactionModel, + ).toBeNull(); + }); +}); + +describe("parseImageLimit", () => { + it("parses a non-negative integer", () => { + expect(parseImageLimit("5")).toEqual({ ok: true, value: 5 }); + expect(parseImageLimit("0")).toEqual({ ok: true, value: 0 }); + }); + + it("floors a fractional value", () => { + expect(parseImageLimit("7.9")).toEqual({ ok: true, value: 7 }); + }); + + it("trims whitespace", () => { + expect(parseImageLimit(" 12 ")).toEqual({ ok: true, value: 12 }); + }); + + it("errors on empty input", () => { + expect(parseImageLimit("")).toEqual({ ok: false, error: "Enter a number." }); + }); + + it("errors on non-numeric input", () => { + expect(parseImageLimit("lots").ok).toBe(false); + }); + + it("errors on a negative value", () => { + expect(parseImageLimit("-1").ok).toBe(false); + }); + + it("errors when above the max", () => { + expect(parseImageLimit(String(MAX_IMAGE_LIMIT + 1)).ok).toBe(false); + }); + + it("accepts the max", () => { + expect(parseImageLimit(String(MAX_IMAGE_LIMIT))).toEqual({ ok: true, value: MAX_IMAGE_LIMIT }); + }); +}); + +describe("imageLimitChanged", () => { + it("is true when the typed value differs after normalization", () => { + expect(imageLimitChanged("5", 10)).toBe(true); + }); + + it("is false when equal", () => { + expect(imageLimitChanged("10", 10)).toBe(false); + }); + + it("is false for invalid input", () => { + expect(imageLimitChanged("abc", 10)).toBe(false); + expect(imageLimitChanged("", 10)).toBe(false); + }); + + it("compares after flooring", () => { + expect(imageLimitChanged("7.9", 7)).toBe(false); + }); +}); + +describe("compactionModelOptions", () => { + const modelInfo: Record<string, ModelMetadata> = { + "kimi/k2": { vision: true }, + "kimi/k1.5": { vision: true }, + "umans/glm-5.2": { vision: false }, + "openai/gpt-4": { contextWindow: 128000 }, + }; + + it("includes the Auto option first", () => { + const opts = compactionModelOptions([], {}); + expect(opts).toHaveLength(1); + expect(opts[0]?.auto).toBe(true); + expect(opts[0]?.value).toBe(AUTO_COMPACTION_MODEL); + }); + + it("includes only vision-capable models, in catalog order", () => { + const models = ["umans/glm-5.2", "kimi/k2", "openai/gpt-4", "kimi/k1.5"]; + const opts = compactionModelOptions(models, modelInfo); + expect(opts.map((o) => o.label)).toEqual(["Auto (server-selected)", "kimi/k2", "kimi/k1.5"]); + }); + + it("excludes non-vision models even with metadata present", () => { + const opts = compactionModelOptions(["umans/glm-5.2"], modelInfo); + expect(opts).toHaveLength(1); // only Auto + }); +}); + +describe("compactionModel value round-trip", () => { + it("selectedCompactionValue maps null to the auto sentinel", () => { + expect(selectedCompactionValue(null)).toBe(AUTO_COMPACTION_MODEL); + }); + + it("selectedCompactionValue maps a model name to itself", () => { + expect(selectedCompactionValue("kimi/k2")).toBe("kimi/k2"); + }); + + it("compactionModelFromValue maps the auto sentinel back to null", () => { + expect(compactionModelFromValue(AUTO_COMPACTION_MODEL)).toBeNull(); + }); + + it("compactionModelFromValue maps a model name to itself", () => { + expect(compactionModelFromValue("kimi/k2")).toBe("kimi/k2"); + }); +}); + +describe("compactionModelChanged", () => { + it("is true when the value maps to a different model", () => { + expect(compactionModelChanged("kimi/k2", null)).toBe(true); + expect(compactionModelChanged(AUTO_COMPACTION_MODEL, "kimi/k2")).toBe(true); + }); + + it("is false when equal (null vs auto, or same model)", () => { + expect(compactionModelChanged(AUTO_COMPACTION_MODEL, null)).toBe(false); + expect(compactionModelChanged("kimi/k2", "kimi/k2")).toBe(false); + }); +}); + +describe("imageLimitLabel", () => { + it("labels null as loading", () => { + expect(imageLimitLabel(null)).toBe("Loading…"); + }); + + it("labels 0 as disabled", () => { + expect(imageLimitLabel(0)).toBe("0 (compaction disabled)"); + }); + + it("labels the default with (default)", () => { + expect(imageLimitLabel(DEFAULT_IMAGE_LIMIT)).toBe(`${DEFAULT_IMAGE_LIMIT} (default)`); + }); + + it("labels other values plainly", () => { + expect(imageLimitLabel(7)).toBe("7"); + }); +}); diff --git a/src/features/vision/logic/view-model.ts b/src/features/vision/logic/view-model.ts new file mode 100644 index 0000000..97a7093 --- /dev/null +++ b/src/features/vision/logic/view-model.ts @@ -0,0 +1,189 @@ +import type { ModelMetadata } from "@dispatch/transport-contract"; +import { isVisionModel } from "../../chat"; + +/** + * Pure core for the vision settings feature — zero DOM, zero effects, zero Svelte. + * + * The global vision configuration (`GET`/`PUT /settings/vision`) controls image + * compaction: how many native images a vision model keeps per turn before the + * oldest are transcribed to text (`imageLimit`), and which vision model does the + * transcribing (`compactionModel`, null = auto). This module is the view-model + * seam: typed settings, parse/validate, dirty-check, network normalization, and + * the vision-capable model option list. The composition root adapts the store's + * HTTP calls to the injected ports. + */ + +// ── Types (owned locally — consumer-defines-port; the contract shapes are a +// plain REST surface not in a shared contract package version bump, mirroring +// the heartbeat/mcp pattern). If the backend promotes these to a shared +// package, swap the local types for the imports + re-mirror. ────────────── + +/** The global vision settings (mirrors `VisionSettingsResponse`). */ +export interface VisionSettings { + /** Max native images per turn (default 10); 0 disables compaction. */ + readonly imageLimit: number; + /** Which model transcribes old images (`<key>/<model>`), or null = auto. */ + readonly compactionModel: string | null; +} + +/** A partial update (mirrors `SetVisionSettingsRequest`). */ +export interface VisionSettingsPatch { + readonly imageLimit?: number; + readonly compactionModel?: string | null; +} + +// ── Injected ports (consumer-defines-port). ─────────────────────────────────── + +/** Outcome of loading the vision settings. */ +export type LoadVisionSettingsResult = + | { readonly ok: true; readonly settings: VisionSettings } + | { readonly ok: false; readonly error: string }; + +/** Outcome of saving a partial vision-settings update. */ +export type SaveVisionSettingsResult = + | { readonly ok: true; readonly settings: VisionSettings } + | { readonly ok: false; readonly error: string }; + +export type LoadVisionSettings = () => Promise<LoadVisionSettingsResult>; +export type SaveVisionSettings = (patch: VisionSettingsPatch) => Promise<SaveVisionSettingsResult>; + +// ── Constants ──────────────────────────────────────────────────────────────── + +/** The backend's default image limit when none is persisted. */ +export const DEFAULT_IMAGE_LIMIT = 10; +/** Upper bound for the image limit input (defensive — the backend owns real clamping). */ +export const MAX_IMAGE_LIMIT = 1000; + +// ── Network normalization (pure; coerces untyped JSON at the seam). ────────── + +/** + * Coerce an untyped JSON body (from `GET /settings/vision`) into a valid + * `VisionSettings`. A malformed/partial response can never crash the renderer: + * `imageLimit` falls back to the default; `compactionModel` coerces to null. + */ +export function normalizeVisionSettings(body: unknown): VisionSettings { + if (body === null || typeof body !== "object") { + return { imageLimit: DEFAULT_IMAGE_LIMIT, compactionModel: null }; + } + const raw = body as Record<string, unknown>; + const limitRaw = raw.imageLimit; + const imageLimit = + typeof limitRaw === "number" && Number.isFinite(limitRaw) && limitRaw >= 0 + ? Math.floor(limitRaw) + : DEFAULT_IMAGE_LIMIT; + const modelRaw = raw.compactionModel; + const compactionModel = typeof modelRaw === "string" && modelRaw.length > 0 ? modelRaw : null; + return { imageLimit, compactionModel }; +} + +// ── imageLimit parse / validate ─────────────────────────────────────────────── + +/** Result of parsing a typed image-limit string. */ +export type ImageLimitParse = + | { readonly ok: true; readonly value: number } + | { readonly ok: false; readonly error: string }; + +/** + * Parse a typed image-limit string into a non-negative integer (floored, + * clamped to [0, MAX_IMAGE_LIMIT]). Empty or non-numeric input is an ERROR + * (so the UI can disable submit + message), NOT a silent default. + */ +export function parseImageLimit(raw: string): ImageLimitParse { + const trimmed = raw.trim(); + if (trimmed.length === 0) { + return { ok: false, error: "Enter a number." }; + } + const n = Number(trimmed); + if (!Number.isFinite(n) || n < 0) { + return { ok: false, error: "Must be 0 or a positive number." }; + } + const floored = Math.floor(n); + if (floored > MAX_IMAGE_LIMIT) { + return { ok: false, error: `Must be at most ${MAX_IMAGE_LIMIT}.` }; + } + return { ok: true, value: floored }; +} + +/** + * Whether saving `typed` would change the `current` image limit. A no-op save + * (empty/invalid, or equal) should be disabled. This is the dirty-check for the + * input — it must NOT mutate or clamp, only compare. + */ +export function imageLimitChanged(typed: string, current: number): boolean { + const parsed = parseImageLimit(typed); + if (!parsed.ok) return false; + return parsed.value !== current; +} + +// ── compactionModel option list ─────────────────────────────────────────────── + +/** + * The sentinel value for the "Auto" option (null compactionModel — the server + * auto-selects a vision model). Used as the `<option value>` for the auto row. + */ +export const AUTO_COMPACTION_MODEL = "__auto__"; + +/** A selectable compaction-model option. */ +export interface CompactionModelOption { + /** The value to send (`<key>/<model>`, or `AUTO_COMPACTION_MODEL` for auto). */ + readonly value: string; + /** The human-readable label. */ + readonly label: string; + /** Whether this is the "Auto" sentinel. */ + readonly auto: boolean; +} + +/** + * Build the compaction-model dropdown options: the "Auto" entry (null) plus + * every vision-capable model from the catalog (those with + * `modelInfo[name].vision === true`), in catalog order. Non-vision models are + * excluded — they cannot transcribe images. + */ +export function compactionModelOptions( + models: readonly string[], + modelInfo: Readonly<Record<string, ModelMetadata>>, +): CompactionModelOption[] { + const options: CompactionModelOption[] = [ + { value: AUTO_COMPACTION_MODEL, label: "Auto (server-selected)", auto: true }, + ]; + for (const name of models) { + if (isVisionModel(modelInfo, name)) { + options.push({ value: name, label: name, auto: false }); + } + } + return options; +} + +/** + * The `<option value>` to mark selected for the current `compactionModel`: + * `AUTO_COMPACTION_MODEL` when null (auto), else the model name itself. + */ +export function selectedCompactionValue(compactionModel: string | null): string { + return compactionModel ?? AUTO_COMPACTION_MODEL; +} + +/** + * Convert a selected `<option value>` back into a `VisionSettingsPatch` + * `compactionModel` (the auto sentinel → null). Returns the patch alone so the + * caller can merge it with other fields. + */ +export function compactionModelFromValue(value: string): string | null { + return value === AUTO_COMPACTION_MODEL ? null : value; +} + +/** + * Whether choosing `value` would change the current `compactionModel`. + */ +export function compactionModelChanged(value: string, current: string | null): boolean { + return compactionModelFromValue(value) !== current; +} + +// ── Labels ──────────────────────────────────────────────────────────────────── + +/** A human-readable label for the current image limit (for the status line). */ +export function imageLimitLabel(imageLimit: number | null): string { + if (imageLimit === null) return "Loading…"; + if (imageLimit === 0) return "0 (compaction disabled)"; + if (imageLimit === DEFAULT_IMAGE_LIMIT) return `${imageLimit} (default)`; + return String(imageLimit); +} diff --git a/src/features/vision/ui/VisionSettingsView.svelte b/src/features/vision/ui/VisionSettingsView.svelte new file mode 100644 index 0000000..2b4ebba --- /dev/null +++ b/src/features/vision/ui/VisionSettingsView.svelte @@ -0,0 +1,192 @@ +<script lang="ts"> + import type { ModelMetadata } from "@dispatch/transport-contract"; + import { + compactionModelChanged, + compactionModelFromValue, + compactionModelOptions, + DEFAULT_IMAGE_LIMIT, + imageLimitChanged, + imageLimitLabel, + parseImageLimit, + selectedCompactionValue, + type LoadVisionSettings, + type SaveVisionSettings, + type VisionSettings, + } from "../logic/view-model"; + + let { + models, + modelInfo = {}, + load, + save, + }: { + /** The model catalog (`GET /models` `models`) — for the compaction-model dropdown. */ + models: readonly string[]; + /** Per-model metadata — to filter the dropdown to vision-capable models. */ + modelInfo?: Readonly<Record<string, ModelMetadata>>; + /** Load the global vision settings (`GET /settings/vision`). */ + load: LoadVisionSettings; + /** Save a partial vision-settings update (`PUT /settings/vision`). */ + save: SaveVisionSettings; + } = $props(); + + let settings = $state<VisionSettings | null>(null); + let loadError = $state<string | null>(null); + + // imageLimit input state. + let imageLimitInput = $state(""); + let savingImageLimit = $state(false); + let imageLimitError = $state<string | null>(null); + let imageLimitSaved = $state(false); + + // compactionModel select state. + let compactionSaving = $state(false); + let compactionError = $state<string | null>(null); + let compactionSaved = $state(false); + + // Load on mount. + $effect(() => { + void refresh(); + }); + + async function refresh(): Promise<void> { + const result = await load(); + if (result.ok) { + settings = result.settings; + imageLimitInput = String(result.settings.imageLimit); + loadError = null; + imageLimitError = null; + imageLimitSaved = false; + compactionError = null; + compactionSaved = false; + } else { + loadError = result.error; + } + } + + const options = $derived(compactionModelOptions(models, modelInfo)); + const selectedCompaction = $derived( + settings ? selectedCompactionValue(settings.compactionModel) : selectedCompactionValue(null), + ); + const limitLabel = $derived(imageLimitLabel(settings?.imageLimit ?? null)); + + const canSaveImageLimit = $derived( + settings !== null && imageLimitChanged(imageLimitInput, settings.imageLimit), + ); + + async function handleSaveImageLimit(): Promise<void> { + if (settings === null || savingImageLimit) return; + const parsed = parseImageLimit(imageLimitInput); + if (!parsed.ok) { + imageLimitError = parsed.error; + imageLimitSaved = false; + return; + } + savingImageLimit = true; + imageLimitError = null; + imageLimitSaved = false; + const result = await save({ imageLimit: parsed.value }); + savingImageLimit = false; + if (result.ok) { + settings = result.settings; + imageLimitInput = String(result.settings.imageLimit); + imageLimitSaved = true; + } else { + imageLimitError = result.error; + } + } + + async function handleCompactionChange(e: Event): Promise<void> { + if (settings === null || compactionSaving) return; + const value = (e.currentTarget as HTMLSelectElement).value; + if (!compactionModelChanged(value, settings.compactionModel)) return; + compactionSaving = true; + compactionError = null; + compactionSaved = false; + const result = await save({ compactionModel: compactionModelFromValue(value) }); + compactionSaving = false; + if (result.ok) { + settings = result.settings; + compactionSaved = true; + } else { + compactionError = result.error; + } + } +</script> + +<div class="flex flex-col gap-3"> + {#if loadError} + <p class="text-xs text-error">{loadError}</p> + {/if} + + {#if settings === null && !loadError} + <p class="text-xs opacity-60">Loading vision settings…</p> + {:else if settings !== null} + <!-- imageLimit --> + <section class="flex flex-col gap-1"> + <span class="text-xs font-semibold uppercase opacity-60">Image limit</span> + <div class="flex items-center gap-2"> + <input + type="text" + inputmode="numeric" + class="input input-bordered input-sm w-24" + placeholder={String(DEFAULT_IMAGE_LIMIT)} + bind:value={imageLimitInput} + disabled={savingImageLimit} + aria-label="Image limit (max native images per turn)" + /> + <button + type="button" + class="btn btn-sm btn-outline" + disabled={!canSaveImageLimit || savingImageLimit} + onclick={handleSaveImageLimit} + > + {#if savingImageLimit} + <span class="loading loading-spinner loading-xs"></span> + Saving… + {:else} + Save + {/if} + </button> + </div> + <p class="text-xs opacity-50"> + Current: {limitLabel} + <br /> + Max native images per turn before the oldest are transcribed to text. + 0 disables compaction. Default is {DEFAULT_IMAGE_LIMIT}. + </p> + {#if imageLimitError} + <p class="text-xs text-error">{imageLimitError}</p> + {:else if imageLimitSaved} + <p class="text-xs text-success">Saved.</p> + {/if} + </section> + + <!-- compactionModel --> + <section class="flex flex-col gap-1"> + <span class="text-xs font-semibold uppercase opacity-60">Compaction model</span> + <select + class="select select-bordered select-sm w-full" + value={selectedCompaction} + disabled={compactionSaving} + onchange={handleCompactionChange} + aria-label="Compaction model (which vision model transcribes old images)" + > + {#each options as opt (opt.value)} + <option value={opt.value}>{opt.label}</option> + {/each} + </select> + {#if compactionSaving} + <p class="text-xs opacity-60">Saving…</p> + {/if} + <p class="text-xs opacity-50"> + The vision-capable model that transcribes old images to text. "Auto" lets the server choose. + </p> + {#if compactionError} + <p class="text-xs text-error">{compactionError}</p> + {:else if compactionSaved} + <p class="text-xs text-success">Saved.</p> + {/if} + </section> + {/if} +</div> diff --git a/src/features/vision/ui/VisionSettingsView.test.ts b/src/features/vision/ui/VisionSettingsView.test.ts new file mode 100644 index 0000000..48afc71 --- /dev/null +++ b/src/features/vision/ui/VisionSettingsView.test.ts @@ -0,0 +1,241 @@ +import { render, screen } from "@testing-library/svelte"; +import userEvent from "@testing-library/user-event"; +import { describe, expect, it, vi } from "vitest"; +import type { + LoadVisionSettingsResult, + SaveVisionSettingsResult, + VisionSettings, +} from "../logic/view-model"; +import VisionSettingsView from "./VisionSettingsView.svelte"; + +const SETTINGS: VisionSettings = { imageLimit: 10, compactionModel: null }; + +function fakeLoad(settings: VisionSettings = SETTINGS): { + calls: number; + impl: () => Promise<LoadVisionSettingsResult>; +} { + let calls = 0; + return { + get calls() { + return calls; + }, + impl: async () => { + calls += 1; + return { ok: true, settings }; + }, + }; +} + +function fakeSaveOk(): { + patches: object[]; + impl: (patch: object) => Promise<SaveVisionSettingsResult>; +} { + const patches: object[] = []; + return { + get patches() { + return patches; + }, + impl: async (patch) => { + patches.push(patch); + // Merge into the current settings to simulate the server echo. + const next: VisionSettings = { + imageLimit: + "imageLimit" in patch ? (patch as VisionSettings).imageLimit : SETTINGS.imageLimit, + compactionModel: + "compactionModel" in patch + ? (patch as VisionSettings).compactionModel + : SETTINGS.compactionModel, + }; + return { ok: true, settings: next }; + }, + }; +} + +describe("VisionSettingsView", () => { + it("loads settings on mount and seeds the imageLimit input", async () => { + const load = fakeLoad({ imageLimit: 7, compactionModel: "kimi/k2" }); + render(VisionSettingsView, { + props: { + models: ["kimi/k2"], + modelInfo: { "kimi/k2": { vision: true } }, + load: load.impl, + save: fakeSaveOk().impl, + }, + }); + + await vi.waitFor(() => { + expect(screen.getByLabelText(/Image limit/)).toHaveValue("7"); + }); + // "Auto" is selected (compactionModel was kimi/k2 here actually) + expect(screen.getByLabelText(/Compaction model/)).toHaveValue("kimi/k2"); + }); + + it("disables Save until the imageLimit input differs", async () => { + const load = fakeLoad(); + const save = fakeSaveOk(); + const user = userEvent.setup(); + render(VisionSettingsView, { + props: { models: [], modelInfo: {}, load: load.impl, save: save.impl }, + }); + + await vi.waitFor(() => { + expect(screen.getByRole("button", { name: "Save" })).toBeDisabled(); + }); + + const input = screen.getByLabelText(/Image limit/); + await user.clear(input); + await user.type(input, "5"); + expect(screen.getByRole("button", { name: "Save" })).toBeEnabled(); + }); + + it("saves the imageLimit on click and confirms", async () => { + const load = fakeLoad(); + const save = fakeSaveOk(); + const user = userEvent.setup(); + render(VisionSettingsView, { + props: { models: [], modelInfo: {}, load: load.impl, save: save.impl }, + }); + + await vi.waitFor(() => { + expect(screen.getByLabelText(/Image limit/)).toHaveValue("10"); + }); + + const input = screen.getByLabelText(/Image limit/); + await user.clear(input); + await user.type(input, "3"); + await user.click(screen.getByRole("button", { name: "Save" })); + + await vi.waitFor(() => { + expect(save.patches).toEqual([{ imageLimit: 3 }]); + }); + expect(screen.getByText(/Saved/i)).toBeInTheDocument(); + }); + + it("shows an error for a non-numeric imageLimit on save", async () => { + const load = fakeLoad(); + const save = fakeSaveOk(); + const user = userEvent.setup(); + render(VisionSettingsView, { + props: { models: [], modelInfo: {}, load: load.impl, save: save.impl }, + }); + + await vi.waitFor(() => { + expect(screen.getByLabelText(/Image limit/)).toHaveValue("10"); + }); + + const input = screen.getByLabelText(/Image limit/); + await user.clear(input); + await user.type(input, "abc"); + // Save is disabled for invalid input, so no save fires; the error surfaces + // only on a submit attempt — but the button is disabled, so just assert that. + expect(screen.getByRole("button", { name: "Save" })).toBeDisabled(); + expect(save.patches).toEqual([]); + }); + + it("renders the compaction-model dropdown with Auto + vision-capable models", async () => { + const load = fakeLoad(); + render(VisionSettingsView, { + props: { + models: ["kimi/k2", "umans/glm-5.2", "kimi/k1.5"], + modelInfo: { + "kimi/k2": { vision: true }, + "kimi/k1.5": { vision: true }, + "umans/glm-5.2": { vision: false }, + }, + load: load.impl, + save: fakeSaveOk().impl, + }, + }); + + await vi.waitFor(() => { + expect(screen.getByLabelText(/Compaction model/)).toBeInTheDocument(); + }); + const select = screen.getByLabelText(/Compaction model/) as HTMLSelectElement; + const optionTexts = Array.from(select.options).map((o) => o.textContent ?? ""); + expect(optionTexts).toEqual(["Auto (server-selected)", "kimi/k2", "kimi/k1.5"]); + // Non-vision glm-5.2 is excluded. + expect(optionTexts.some((t) => t.includes("glm-5.2"))).toBe(false); + }); + + it("saves the compactionModel on change (Auto → a vision model)", async () => { + const load = fakeLoad({ imageLimit: 10, compactionModel: null }); + const save = fakeSaveOk(); + const user = userEvent.setup(); + render(VisionSettingsView, { + props: { + models: ["kimi/k2"], + modelInfo: { "kimi/k2": { vision: true } }, + load: load.impl, + save: save.impl, + }, + }); + + await vi.waitFor(() => { + expect(screen.getByLabelText(/Compaction model/)).toBeInTheDocument(); + }); + + await user.selectOptions(screen.getByLabelText(/Compaction model/), "kimi/k2"); + + await vi.waitFor(() => { + expect(save.patches).toEqual([{ compactionModel: "kimi/k2" }]); + }); + expect(screen.getByText(/Saved/i)).toBeInTheDocument(); + }); + + it("saves null (Auto) when the auto option is chosen", async () => { + const load = fakeLoad({ imageLimit: 10, compactionModel: "kimi/k2" }); + const save = fakeSaveOk(); + const user = userEvent.setup(); + render(VisionSettingsView, { + props: { + models: ["kimi/k2"], + modelInfo: { "kimi/k2": { vision: true } }, + load: load.impl, + save: save.impl, + }, + }); + + await vi.waitFor(() => { + expect(screen.getByLabelText(/Compaction model/)).toHaveValue("kimi/k2"); + }); + + await user.selectOptions(screen.getByLabelText(/Compaction model/), "__auto__"); + + await vi.waitFor(() => { + expect(save.patches).toEqual([{ compactionModel: null }]); + }); + }); + + it("surfaces a load error", async () => { + const load = vi.fn(async () => ({ ok: false, error: "vision unavailable" }) as const); + render(VisionSettingsView, { + props: { models: [], modelInfo: {}, load, save: fakeSaveOk().impl }, + }); + + await vi.waitFor(() => { + expect(screen.getByText("vision unavailable")).toBeInTheDocument(); + }); + }); + + it("surfaces a save error", async () => { + const load = fakeLoad(); + const save = vi.fn(async () => ({ ok: false, error: "boom" }) as const); + const user = userEvent.setup(); + render(VisionSettingsView, { + props: { models: [], modelInfo: {}, load: load.impl, save }, + }); + + await vi.waitFor(() => { + expect(screen.getByLabelText(/Image limit/)).toHaveValue("10"); + }); + + const input = screen.getByLabelText(/Image limit/); + await user.clear(input); + await user.type(input, "3"); + await user.click(screen.getByRole("button", { name: "Save" })); + + await vi.waitFor(() => { + expect(screen.getByText("boom")).toBeInTheDocument(); + }); + }); +}); |
