diff options
| author | Adam Malczewski <[email protected]> | 2026-06-03 13:02:15 +0900 |
|---|---|---|
| committer | Adam Malczewski <[email protected]> | 2026-06-03 13:02:15 +0900 |
| commit | e87e6b39285c8001045d1ebdac873b182c0f7868 (patch) | |
| tree | 27003852f7b182fd65c6ad762784aa5fcf839ebc /packages/api/src/app.ts | |
| parent | ae672fd4f5542a2c217cf97657bf81eeebdaabbd (diff) | |
| download | dispatch-e87e6b39285c8001045d1ebdac873b182c0f7868.tar.gz dispatch-e87e6b39285c8001045d1ebdac873b182c0f7868.zip | |
feat: prompt cache warming for idle tabs
Keep a tab's provider prompt-cache warm while idle by periodically replaying
the exact cached conversation prefix plus a single trivial throwaway turn,
resetting the provider's ~5-min cache TTL so the user's next real message hits
a warm cache.
Backend:
- Agent.warmCache(history): extracts buildLlmContext() shared with run(), then
re-sends the identical system+tools+history prefix (same Anthropic
cache_control breakpoints) plus a 'reply with just a .' probe turn via
toolChoice:none. Returns the request usage; mutates no history, emits/persists
nothing.
- AgentManager.warmCacheForTab(): resolves the same agent the next real turn
would use, replays the FULL genuine history, refuses while a turn is running.
- POST /chat/warm: returns ONLY the warming request's usage (never persisted,
never folded into the real usage aggregate).
Frontend:
- cache-warming.svelte.ts store: per-tab 4-min repeating idle timer with
countdown, warming-specific last-request cache %, and error capture. Arms on
turn end, pauses during a turn, disables+resets on a real user message.
- cache-warm-storage.ts: per-tab localStorage persistence of the toggle.
- Lifecycle hooks wired into tabs.svelte.ts (status/statuses/sendMessage/
hydrate/create/open/close).
- ModelSelector: bottom-of-panel checkbox + debug strip (last-% / countdown /
error), shown only when enabled. Warming cache data never touches the real
Cache Rate view.
Tests: core warmCache (5), api warm route (3) + warmCacheForTab (3), frontend
store (12) + storage (10). check / test (779) / frontend build / typecheck all
green.
Diffstat (limited to 'packages/api/src/app.ts')
| -rw-r--r-- | packages/api/src/app.ts | 37 |
1 files changed, 37 insertions, 0 deletions
diff --git a/packages/api/src/app.ts b/packages/api/src/app.ts index 2f4e538..a957da7 100644 --- a/packages/api/src/app.ts +++ b/packages/api/src/app.ts @@ -224,6 +224,43 @@ app.post("/chat/stop", async (c) => { return c.json({ success: true }); }); +// Prompt-cache WARMING (see AgentManager.warmCacheForTab / Agent.warmCache). +// +// Replays the tab's exact cached prefix + one trivial throwaway turn so the +// provider's ~5-min prompt-cache TTL is refreshed while the tab sits idle. +// The frontend's cache-warming timer drives this every ~4 minutes. The +// warming request is NEVER persisted, NEVER emitted, and NEVER folded into the +// real usage aggregate — we return ONLY its `usage` so the UI can show a +// warming-specific "last request" cache rate without polluting the real +// Cache Rate metric. Returns 409 when the tab is mid-turn (caller also gates). +app.post("/chat/warm", async (c) => { + const body = await c.req.json<{ + tabId?: unknown; + keyId?: unknown; + modelId?: unknown; + agentModels?: unknown; + }>(); + const { tabId } = body; + if (typeof tabId !== "string" || tabId.trim() === "") { + return c.json({ error: "tabId must be a non-empty string" }, 400); + } + const keyId = typeof body.keyId === "string" ? body.keyId : undefined; + const modelId = typeof body.modelId === "string" ? body.modelId : undefined; + const agentModels = sanitizeAgentModels(body.agentModels); + + const result = await agentManager.warmCacheForTab(tabId, { + ...(keyId ? { keyId } : {}), + ...(modelId ? { modelId } : {}), + ...(agentModels ? { agentModels } : {}), + }); + if (!result.ok) { + // "tab is generating" is an expected race (not a server fault) → 409. + const status = result.error === "tab is generating" ? 409 : 500; + return c.json({ error: result.error }, status); + } + return c.json({ usage: result.usage }); +}); + app.route("/skills", skillsRoutes); app.route("/models", modelsRoutes); app.route("/tabs", tabsRoutes); |
