diff options
Diffstat (limited to 'plan-chunk-refactor.md')
| -rw-r--r-- | plan-chunk-refactor.md | 245 |
1 files changed, 245 insertions, 0 deletions
diff --git a/plan-chunk-refactor.md b/plan-chunk-refactor.md new file mode 100644 index 0000000..0a60237 --- /dev/null +++ b/plan-chunk-refactor.md @@ -0,0 +1,245 @@ +# Plan: Chunk-Based Message Refactor + +## Goal + +Replace the current `content: ContentSegment[]` + separate `thinking: string` representation with a single ordered `chunks: Chunk[]` list per message. Preserve the actual temporal ordering of text, reasoning, tool calls, system notices, and errors as they arrive from the model — losing none of that information to flat-string accumulation. + +This is foundational work for downstream features (editable history, resumable mid-generation chats, structured truncation sentinels) that all assume an honest, lossless representation of a turn. + +Beta software — **no backward compatibility required**. The existing `messages` and `tabs` rows will be destroyed. Settings, keys, credentials, and other non-chat-history state will be preserved. + +## Final Design + +### Chunk union + +| Type | Body | Opens on | Closes on | Sent to LLM? | +|---|---|---|---|---| +| `text` | `text: string` | First `text-delta` after non-text. Consecutive `text-delta` events append. | `reasoning-delta`, `tool-call`, `error`, system event, `done`. | Yes (as `text` part). | +| `thinking` | `text: string` | First `reasoning-delta` after non-thinking. Consecutive `reasoning-delta` events append. | `text-delta`, `tool-call`, `error`, system event, `done`. | Yes (as `reasoning` part — handles Claude's `interleaved-thinking-2025-05-14`). | +| `tool-batch` | `calls: Array<{ id, name, arguments, result?, isError?, shellOutput? }>` | First `tool-call` after non-tool. Consecutive `tool-call` events append a new entry to `calls`. | Any non-tool event. | Yes (each call/result as `tool-call` / `tool-result` parts). | +| `error` | `message: string`, `statusCode?: number` | An `error` event. Always its own chunk. No coalescing. | Single-event. | No (the turn ended anyway). | +| `system` | `text: string`, `kind: "notice" \| "model-changed" \| "config-reload" \| "cancelled"` | A system event. Always its own chunk. No coalescing. | Single-event. | **No — stripped in `toCoreMessages`.** | + +### Message roles + +`user | assistant | system` + +- `user` and `assistant` messages can contain any chunk types. +- `system` role messages exist for system events that fire **outside an active assistant turn**. They contain only `system` chunks. They are skipped entirely by `toCoreMessages`. + +### System event routing + +When a system event arrives: + +1. Active assistant turn in flight → append a `system` chunk to that message's chunks at its current position. +2. No turn in flight; most recent message is `role: "system"` → append a `system` chunk to that message. +3. No turn in flight; most recent message is anything else → create a new `role: "system"` message containing one `system` chunk. + +### `toCoreMessages` rebuild rules + +- Iterate messages in seq order. +- Skip `role: "system"` messages entirely. +- For each message, iterate `chunks` and emit AI SDK parts: + - `text` → `{ type: "text", text }` + - `thinking` → `{ type: "reasoning", text }` + - `tool-batch` → one `{ type: "tool-call" }` per entry (and tool-result parts in the following `tool` message, same as current logic) + - `error` → skip + - `system` → skip + +### Example turn + +User clicks Send, model thinks, says something, hits a rate limit (auto-switches model), thinks more, calls two tools, says something, errors out: + +``` +Message #N (role=user): + chunks: [{ type: "text", text: "explain X" }] + +Message #N+1 (role=assistant): + chunks: [ + { type: "thinking", text: "I should..." }, + { type: "text", text: "Sure, here's the gist..." }, + { type: "system", kind: "model-changed", text: "Switched to Sonnet 4 (rate limit)" }, + { type: "thinking", text: "Now I need to look at..." }, + { type: "tool-batch", calls: [ + { id: "1", name: "read_file", ..., result: "..." }, + { id: "2", name: "list_files", ..., result: "..." } + ]}, + { type: "text", text: "Looking at the file..." }, + { type: "error", message: "Network error", statusCode: 503 } + ] +``` + +## Database cleanup + +**Live DB:** `~/.local/share/dispatch/dispatch.db` (XDG: `$XDG_DATA_HOME/dispatch/dispatch.db`) + +Current size: ~122 MB. Tables and row counts as of this plan: + +| Table | Rows | Disposition | +|---|---|---| +| `api_keys` | 7 | **Preserve** — user-imported API keys | +| `credentials` | 2 | **Preserve** — OAuth credentials | +| `settings` | 11 | **Preserve** — app settings | +| `usage_cache` | 2 | **Preserve** — usage report cache | +| `wake_schedule` | 5 | **Preserve** — wake schedule | +| `messages` | 524 | **Delete all** — chat history | +| `tabs` | 361 | **Delete all** — chat sessions | + +### Proposed cleanup statements + +```sql +-- Order matters: messages references tabs.id via foreign key (ON in WAL mode) +DELETE FROM messages; +DELETE FROM tabs; +VACUUM; +``` + +Then, in the live schema, drop the `thinking` column from `messages` (or recreate the table without it). SQLite has historically been fussy about `DROP COLUMN`; modern SQLite (3.35+) supports it directly: + +```sql +ALTER TABLE messages DROP COLUMN thinking; +``` + +If the installed SQLite is older, fall back to the table-rebuild pattern (create new table without the column, copy rows, swap names, drop old). Given we just `DELETE FROM messages`, the copy step is a no-op — we can just `DROP TABLE messages` and recreate via the application's `CREATE TABLE IF NOT EXISTS` on next startup (after the schema change in `db/index.ts`). + +**Preferred sequence:** + +1. Stop any running dispatch processes. +2. Run `DELETE FROM messages; DELETE FROM tabs; VACUUM;` to clear chat history. +3. Update `db/index.ts` to remove the `thinking` column from the `CREATE TABLE messages` statement. +4. Drop the existing messages table so the new schema takes effect on next startup: `DROP TABLE messages; DROP INDEX IF EXISTS idx_messages_tab;` (the app will recreate it on launch). + +Effective end state: same DB file, settings/keys preserved, chat history gone, new schema. + +## Implementation phases + +### Phase 0 — Confirm starting point + +- [ ] `git status` clean (only `wishlist.md` untracked is acceptable — unrelated). +- [ ] No running dispatch processes that hold the DB. + +### Phase 1 — Types + +File: `packages/core/src/types/index.ts` + +- [ ] Define `Chunk` union (5 variants per the table above). +- [ ] Replace `ChatMessage.content: string` + `toolCalls?` + `toolResults?` with `chunks: Chunk[]`. +- [ ] Update `MessageRole` to include `"system"` (currently `"user" | "assistant" | "tool"` — drop `"tool"` since tool messages are now embedded as `tool-batch` chunks). + +File: `packages/frontend/src/lib/types.ts` + +- [ ] Mirror `Chunk` union. +- [ ] Replace `content: ContentSegment[]` + `thinking?: string` with `chunks: Chunk[]`. +- [ ] Drop `ContentSegment` and `ToolCallDisplay` (replaced by chunk variants). + +### Phase 2 — Core helper + unit tests + +New file: `packages/core/src/chunks/append.ts` (or co-located in `agent/`) + +- [ ] Implement `appendEventToChunks(chunks: Chunk[], event: AgentEvent): void` (mutating, returns void). +- [ ] Implement `applySystemEvent(messages: ChatMessage[], event: SystemEvent): void` for the standalone-system-message routing logic. + +Tests: `packages/core/tests/chunks/append.test.ts` + +- [ ] Empty chunks + text-delta → one text chunk with the delta. +- [ ] Two consecutive text-deltas → one text chunk with concatenated text. +- [ ] text-delta then reasoning-delta → two chunks (text, thinking). +- [ ] text-delta then tool-call → two chunks (text, tool-batch with one entry). +- [ ] Two consecutive tool-calls → one tool-batch with two entries. +- [ ] tool-call then tool-call then text → two chunks (tool-batch with 2 entries, text). +- [ ] tool-result arrives → updates matching tool-call entry in the latest tool-batch chunk by id. +- [ ] shell-output arrives → appends to the most recent tool-call's `shellOutput`. +- [ ] error event → opens an error chunk; subsequent events go to new chunks. +- [ ] system event during text run → closes text, opens system, would re-open text on next text-delta. +- [ ] Two consecutive system events → two separate system chunks (no coalescing). +- [ ] Interleaved think → text → think → tool → think → text → 6 chunks in order. + +### Phase 3 — Database schema + cleanup + +File: `packages/core/src/db/index.ts` + +- [ ] Update `CREATE TABLE messages` to drop the `thinking` column. + +Live DB: + +- [ ] Run `DELETE FROM messages; DELETE FROM tabs; VACUUM;` (show user before executing). +- [ ] Drop the `messages` table so the new schema takes effect on next startup. + +File: `packages/core/src/db/messages.ts` + +- [ ] Update `appendMessage()` signature: drop the `thinking` parameter; `content_json` now holds chunks. +- [ ] Update read functions to parse `content_json` as `Chunk[]` directly (no thinking column read). + +### Phase 4 — Agent aggregation + +File: `packages/core/src/agent/agent.ts` + +- [ ] Replace `finalText: string`, `allToolCalls: ToolCall[]`, `allToolResults: ToolResult[]`, and `assistantThinking` (lines 331-333 and the equivalent in agent-manager) with a single `chunks: Chunk[]`. +- [ ] On each event from `result.fullStream`, call `appendEventToChunks(chunks, event)`. +- [ ] On `done`, ship `chunks` in the message payload. +- [ ] Update `toCoreMessages` (lines 20-46): iterate chunks per message, emit AI SDK parts, skip system / error chunks, skip `role: "system"` messages entirely. + +### Phase 5 — Persistence + system event routing + +File: `packages/api/src/agent-manager.ts` + +- [ ] Replace the three-accumulator pattern (lines 893-978) with a single `chunks: Chunk[]` that delegates to `appendEventToChunks`. +- [ ] On `notice` / `model-changed` / `config-reload` / cancel: route through `applySystemEvent` — appends to the in-flight assistant message if one exists, else appends to a `role: "system"` message (creating one if needed). +- [ ] `appendMessage` call: drop the thinking arg; `content_json` is `JSON.stringify(chunks)`. + +### Phase 6 — Frontend store state machine + +File: `packages/frontend/src/lib/tabs.svelte.ts` + +- [ ] Replace per-event-type mutation logic (lines 288-626) with a call to the shared `appendEventToChunks` helper. Import from core if feasible; otherwise duplicate carefully — but prefer import to keep wire-format symmetry guaranteed. +- [ ] Update `openAgentTab` DB-load path (lines 185-199): parse `chunks` directly from `content_json`, no thinking field merging. +- [ ] Update `currentAssistantId` semantics: still tracks the in-progress assistant message. System events that arrive when `currentAssistantId` is null create/append a `role: "system"` message via `applySystemEvent`. + +### Phase 7 — Frontend display + +File: `packages/frontend/src/lib/components/ChatMessage.svelte` + +- [ ] Remove the top-hoisted `{#if message.thinking}` block. +- [ ] Replace the segments loop with `{#each message.chunks as chunk}` and switch on `chunk.type`: + - `text` → `<MarkdownRenderer>` + - `thinking` → collapsible (DaisyUI `collapse`), default-collapsed per `appSettings.autoExpandThinking`, at its actual position in the turn. + - `tool-batch` → render each entry via `<ToolCallDisplay>` (existing component, possibly minor prop tweaks). + - `error` → red-bordered error card with the message + status code. + - `system` → thin separator-style block with the `kind` as a small label and the text. +- [ ] Handle `role: "system"` messages: render as a standalone thin bubble with just the system chunks, no avatar / no actions. + +### Phase 8 — Integration tests + +- [ ] End-to-end: a real streaming run produces the expected chunk shape, including interleaved thinking. +- [ ] Persistence round-trip: write chunks → read back → identical structure. +- [ ] System event during turn ends up in the assistant message at the right position. +- [ ] System event with no turn in flight creates a `role: "system"` message. +- [ ] `toCoreMessages` correctly strips system / error chunks and `role: "system"` messages. + +### Phase 9 — Cleanup + +- [ ] Remove dead code: `ContentSegment` type, `thinking` field references, old per-event accumulator vars. +- [ ] Run `vitest run` across all packages and `tsc --noEmit`. +- [ ] Commit with a focused message; push. + +## Test strategy + +The load-bearing piece is `appendEventToChunks`. If this is correct in isolation, every downstream layer inherits correctness. **Write Phase 2 first and lock its tests in green before touching agent/store/UI.** + +State-machine tests should cover every transition pair in the matrix (text→text, text→thinking, text→tool, text→error, text→system, thinking→text, ...) so that no transition gets accidentally broken later. + +Integration tests can be lighter — they just need to confirm the real wire format flows through the helper end-to-end. + +## Risks and notes + +1. **Frontend store imports from core.** Currently the frontend duplicates some types. Importing `appendEventToChunks` from core is the safest way to guarantee the wire format stays in sync — but requires the build to handle the import. If it doesn't, duplicate the helper carefully and have a test that runs both side by side on the same fixture events to detect drift. + +2. **`ai` SDK `reasoning` events.** Confirmed that `result.fullStream` emits `reasoning` events with `event.textDelta` for Anthropic with the `interleaved-thinking-2025-05-14` beta header. Other providers (OpenAI-compat) emit reasoning via different mechanisms; the middleware in `provider.ts:6-38` already handles those. + +3. **Tool result ordering across multiple steps.** Currently, `tool-result` events arrive at varying times relative to subsequent `text-delta` events depending on whether the AI streams text before vs after acknowledging the result. The new chunk model places results inside the `tool-batch` chunk that holds the matching call — this should be correct regardless of when the result event arrives, as long as the lookup-by-id finds the right batch. + +4. **Coalescing edge case: simultaneous tool calls.** If the AI emits multiple tool-call events with no intervening events (parallel tool use), they all batch together. If a `tool-result` for the first call arrives before the second `tool-call` event, the tool-batch chunk is already open and just absorbs both. No special handling needed. + +5. **Settings preservation.** The `settings` table (`key`, `value` columns) is preserved across the wipe. Verify the dispatch UI doesn't rely on any session-scoped row that lives inside `tabs` (e.g., "last opened tab id") — if so, the user will need to re-pick after the wipe. This is acceptable for beta. + +6. **`MessageRole` change.** Dropping `"tool"` from the role union means any code path that pattern-matches on `role === "tool"` must change. Grep for it before merging. |