diff options
| author | Adam Malczewski <[email protected]> | 2026-06-02 22:50:11 +0900 |
|---|---|---|
| committer | Adam Malczewski <[email protected]> | 2026-06-02 22:50:11 +0900 |
| commit | 66e5d3b105bfd2b34c6f35876bf33dbb3cb9dcae (patch) | |
| tree | c3e039e09c89231f84dfd16f7bbbf8aedcc2dc7d /packages/frontend/src/App.svelte | |
| parent | 4b45d33c256cf580a53054078be6fd7148fa6302 (diff) | |
| download | dispatch-66e5d3b105bfd2b34c6f35876bf33dbb3cb9dcae.tar.gz dispatch-66e5d3b105bfd2b34c6f35876bf33dbb3cb9dcae.zip | |
feat(chat): paste-to-attach images/PDFs with model capability check
Add multimodal image/PDF input to the chat box via clipboard paste, gated by a
graceful per-model capability check.
UX: a pasted image/PDF inserts an inline token (【image:…】 / 【pdf:…】) into the
draft, so attachments have ORDER relative to typed text and can be referenced
positionally. The token is the only handle — deleting it (atomic Backspace/
Delete, or selection overlap) detaches the file; an input-reconciliation safety
net detaches any attachment whose token is no longer intact. No preview strip.
Capability check: resolveModelCapabilities reads models.dev modalities.input
(new GET /models/capabilities, mirrors /context-limit). The input blocks Send
(no tokens spent) only on a definitive 'no'; unknown capability (catalog offline
/ unmapped provider) stays permissive. Attachments require a fresh turn — Send is
blocked while generating and /chat rejects content mid-turn (409).
Attachments are EPHEMERAL: forwarded to the model for the turn via ordered AI SDK
ImagePart/FilePart content, but never persisted (history keeps the text with
[image]/[pdf] markers). Text-only turns serialize byte-identically to before.
Limits (Anthropic-aligned, enforced at paste + re-validated server-side):
PNG/JPEG/WebP/GIF/PDF; image ≤5MB, PDF ≤32MB, ≤20 attachments, ≤32MB total.
core: UserContentPart types, models/attachments validator, capability resolver,
agent.run+toModelMessages thread ordered content. api: /chat content validation +
passthrough. frontend: attachment-tokens helper, ChatInput paste/token/gating,
per-tab staged attachments, App.svelte capability fetch. +44 tests.
Diffstat (limited to 'packages/frontend/src/App.svelte')
| -rw-r--r-- | packages/frontend/src/App.svelte | 55 |
1 files changed, 54 insertions, 1 deletions
diff --git a/packages/frontend/src/App.svelte b/packages/frontend/src/App.svelte index a0b25b7..ae0718e 100644 --- a/packages/frontend/src/App.svelte +++ b/packages/frontend/src/App.svelte @@ -131,6 +131,59 @@ $effect(() => { })(); }); +// ─── Image / PDF capability lookup ───────────────────────────── +// Resolve whether the active model accepts image/pdf INPUT from models.dev (via +// the API), so the chat input can block sending an unsupported attachment +// (no tokens spent) while staying permissive when the capability is unknown. +// `null` = unknown (catalog offline / unsupported provider) → optimistic allow. +let imageSupport = $state<{ image: boolean; pdf: boolean } | null>(null); +const capabilityCache = new Map<string, { image: boolean; pdf: boolean } | null>(); + +$effect(() => { + const tab = tabStore.activeTab; + const keyId = tab?.keyId ?? null; + const modelId = tab?.modelId ?? null; + const provider = keyId ? (modelsData.keys.find((k) => k.id === keyId)?.provider ?? null) : null; + + if (!provider || !modelId) { + imageSupport = null; + return; + } + + const cacheKey = `${provider}/${modelId}`; + if (capabilityCache.has(cacheKey)) { + imageSupport = capabilityCache.get(cacheKey) ?? null; + return; + } + + // Clear immediately so a slow/failed fetch can't leave the PREVIOUS model's + // capability on screen (which could wrongly block/allow this model). + imageSupport = null; + + void (async () => { + try { + const res = await fetch( + `${config.apiBase}/models/capabilities?provider=${encodeURIComponent(provider)}&modelId=${encodeURIComponent(modelId)}`, + ); + if (!res.ok) return; + const data = (await res.json()) as { + capabilities?: { image: boolean; pdf: boolean } | null; + }; + const caps = data.capabilities ?? null; + capabilityCache.set(cacheKey, caps); + const current = tabStore.activeTab; + const currentProvider = current?.keyId + ? (modelsData.keys.find((k) => k.id === current.keyId)?.provider ?? null) + : null; + if (currentProvider === provider && current?.modelId === modelId) { + imageSupport = caps; + } + } catch { + // Leave imageSupport as null (unknown → permissive) on network error. + } + })(); +}); + onMount(() => { // Apply persisted theme (or the shared DEFAULT_THEME if nothing is // stored) so the first paint matches what the Settings panel will @@ -174,7 +227,7 @@ onMount(() => { <div class="flex-1 overflow-hidden"> <ChatPanel /> </div> - <ChatInput {contextLimit} /> + <ChatInput {contextLimit} {imageSupport} /> </div> <!-- Right sidebar: overlay on small screens, inline on large --> |
