Merge branch 'dev' into feature/workspace-star

2 files changed, 587 insertions, 0 deletions

diff --git a/notes/crash-investigation-findings.md b/notes/crash-investigation-findings.md
new file mode 100644
index 0000000..485e02d
--- /dev/null
+++ b/notes/crash-investigation-findings.md
@@ -0,0 +1,256 @@
+# Production Crash Investigation — Findings
+
+> **Definitive post-investigation record.** Supersedes `notes/memory-leak-investigation-handoff.md`
+> (which contained three material errors — see §3). Authored by opencode (umans-glm-5.2) on
+> 2026-06-28, incorporating an independent Gemini review (`crash-review-report.md`) whose pivotal
+> finding (the SSH pool) overturned an earlier wrong conclusion.
+
+**Last updated:** 2026-06-28
+**Status:** Root cause of the **live** (1.3.14) crash is confirmed and fixable in Dispatch code.
+The native segfault is **not** root-caused and may already be fixed by the 1.3.14 upgrade —
+needs observation under real load.
+
+---
+
+## 0. TL;DR
+
+- The production server was crash-looping (~20 restarts on Jun 28). **Two distinct failure modes.**
+- **Live crash (Bun 1.3.14):** `exit-1 "Timed out while waiting for handshake"`. Root cause is a
+  **Dispatch bug** in `packages/ssh/src/pool.ts`: the pooled `ssh2.Client` has **no permanent
+  `'error'` listener** (it's removed after connect by `cleanup()`, pool.ts:266), so a post-connect
+  ssh2 error emits `'error'` with no listener → uncaught → **no `process.on("uncaughtException")`
+  guard in main.ts** → process exit-1. Fixable in Dispatch code; no runtime change needed.
+- **Segfaults:** all on **Bun v1.3.13** (`Bun v1.3.13 (bf2e2cec)` printed at each). **Zero
+  segfaults observed on 1.3.14** so far. Not root-caused; may be fixed by the upgrade.
+- The prior "~2.5 GB/hour memory leak" framing is **wrong** (see §3). Idle memory is flat at 84 MB.
+- **Recovery on restart** is repair-and-seal, not resume: conversations are swept `active→idle`
+  and partial turns reconciled to a consistent state, but in-flight turns are sealed (user re-sends).
+
+---
+
+## 1. The live crash — SSH pool missing error listener (CONFIRMED, Dispatch bug)
+
+### Crash signature (the only crash on the current 1.3.14 binary)
+```
+error: Timed out while waiting for handshake
+      at emitError (node:events:51:13)
+      at <anonymous> (/$bunfs/root/dispatch-server:16:75617)
+Bun v1.3.14 (Linux x64)
+→ Main process exited, code=exited, status=1/FAILURE
+```
+- 09:22:04 JST crash; last telemetry sample (09:21:52) showed `rss=116MB, activeConversations=4`.
+- systemd reported 2.7 GB peak, but the crash was **exit-1 (self-exit), not OOM/SIGKILL** — memory
+  was a symptom, not the kill trigger.
+
+### Root cause chain
+1. The error string `"Timed out while waiting for handshake"` is **ssh2's**, not Bun's TLS. It is
+   hardcoded in `ssh2/lib/client.js:1114`:
+   ```js
+   this._readyTimeout = setTimeout(() => {
+     const err = new Error('Timed out while waiting for handshake');
+     this.emit('error', err);   // ← ssh2 emits 'error' on the ssh2.Client
+     sock.destroy();
+   }, this.config.readyTimeout);
+   ```
+   (Bun's own TLS string is the different `"TLS handshake timeout"` — confirmed via `strings` on
+   the binary.)
+2. In `packages/ssh/src/pool.ts`, `doConnect` attaches `client.on("error", onError)` **only during
+   the connect promise** and removes it in `cleanup()` (pool.ts:266) on success. `buildConnection`
+   (pool.ts:101-175) returns a long-lived `SshConnection` (keepalive 30s, idle-reap 15m) and
+   **never attaches a permanent `'error'` listener** to the pooled client.
+3. So after a successful connect, the `ssh2.Client` sits in the pool with no `'error'` listener. A
+   later ssh2 error (re-handshake/re-key/keepalive timeout emitting the string above) →
+   `emit('error')` → `emitError (node:events)` with no listener → EventEmitter default throws →
+   uncaught.
+4. `packages/host-bin/src/main.ts` has **only `SIGINT`/`SIGTERM` handlers** (main.ts:280-281) —
+   **no `process.on("uncaughtException")` / `process.on("unhandledRejection")`**. So the uncaught
+   error exits the process (exit-1), killing every in-flight turn — not just the one using SSH.
+
+### Why capping concurrency is NOT the fix
+A single pooled SSH connection dropping crashes the server regardless of how many turns are
+concurrent. Capping `activeConversations` would only reduce frequency, not fix the root cause —
+and it cripples a product built around concurrent agents. (The existing provider-concurrency
+limiter, capped at 3-4, already bounds concurrent *provider streams*; SSH connections are
+tool-execution connections and bypass it.)
+
+### The fix (root cause)
+1. **`packages/ssh/src/pool.ts` — `buildConnection`:** attach a permanent `'error'` listener to
+   the `ssh2.Client` that tears down the connection, sets `state.value = "error"` /
+   `state.error = <msg>`, and **logs** (computer alias, error message, connection state) so the
+   failure is observable. Keep the existing connect-time `onError` for the connect promise.
+2. **`packages/host-bin/src/main.ts` — boot:** add `process.on("uncaughtException")` and
+   `process.on("unhandledRejection")` handlers that **log rich detail** (message, stack,
+   `activeConversations` count, `process.memoryUsage()` snapshot, timestamp) so we know *where*
+   they happen, then either survive (for non-fatal) or seal in-flight work + exit gracefully. This
+   is the defense-in-depth that ensures any *future* unhandled error degrades instead of taking
+   down the whole server and every in-flight turn.
+
+---
+
+## 2. The segfaults — NOT root-caused; 1.3.13 only (may be fixed on 1.3.14)
+
+### Crash signature
+```
+panic(main thread): Segmentation fault at address 0x0
+oh no: Bun has crashed. This indicates a bug in Bun, not your code.
+→ Main process exited, code=dumped, status=4/ILL
+```
+- **Every segfault was on Bun v1.3.13** (the log prints `Bun v1.3.13 (bf2e2cec)` at each:
+  00:12, 00:45, 00:57, 08:50, 09:05).
+- Memory peaks ranged widely: **370 MB** (09:05, 1-min uptime) up to 6.2 GB (00:12, 2.5h uptime).
+- The 1.3.14 binary first ran at 09:06 JST. **No segfault has been observed on 1.3.14** — the only
+  1.3.14 crash is the SSH exit-1 above. The current process has been stable idle.
+
+### What this means
+- The 370 MB segfault is **not memory exhaustion** (24 GB cgroup) — it's a native concurrency/race
+  bug in Bun's runtime. No amount of memory bounding prevents a native race.
+- The 1.3.14 upgrade **may** have fixed it; the sample so far is small and mostly idle, so this is
+  **not confirmed**.
+- **A native segfault cannot be caught by `process.on("uncaughtException")`** (that catches JS
+  exceptions, not SIGILL/SIGSEGV). So if the segfault recurs on 1.3.14, the only complete fixes
+  are: leave Bun (port to Node) or root-cause the native trigger (core dump → backtrace → repro →
+  upstream Bun report).
+
+### Plan for the segfault
+1. Enable core-dump capture (`ulimit -c unlimited` / systemd `COREdump`) so the next segfault
+   produces a backtrace rather than a bare `address 0x0`.
+2. Run 1.3.14 under the real orchestrator workload (concurrent backend+frontend+subagent turns).
+3. If no segfault recurs → likely already fixed; close the issue.
+4. If one recurs → root-cause via the backtrace; do NOT paper over it with concurrency caps.
+
+---
+
+## 3. What was WRONG in the prior handoff (corrections)
+
+`notes/memory-leak-investigation-handoff.md` was a careful document but contained three material
+errors that this investigation corrected:
+
+1. **"Telemetry is in journald — `journalctl | grep memory:periodic`."** Wrong. The logger writes
+   to a **journal sink file** (`DISPATCH_JOURNAL=/var/log/dispatch/app.ndjson`, main.ts:139),
+   not stderr/journald. The handoff's grep found nothing *by design* — the data was in the file
+   all along. (Also: the observability-collector that would drain this file into `traces.db` is
+   **disabled in compiled binaries** — main.ts:150 guards on a source path that doesn't exist in
+   prod — so the journal just accumulates and rotates; only `app.ndjson` + `.1` are retained.)
+2. **"The segfault persists on Bun 1.3.14" (handoff hypothesis #3 disproven).** Wrong. All
+   segfaults were on **1.3.13**; no segfault has been seen on 1.3.14. The 1.3.14 upgrade may have
+   fixed it — the handoff's hypothesis #3 is *not* disproven after all.
+3. **"~2.5 GB/hour slow memory leak."** Wrong framing. Telemetry shows **idle is flat at 84 MB**
+   (20+ min, zero reclaimable, zero growth) — there is no background/timer/closure leak. The
+   "leak" was the live working set of concurrent multi-step turns, which is reclaimed when turns
+   end. Moreover, raw context size is never GB-scale: 300k tokens ≈ 1-2 MB, so the gigabytes
+   under load must come from pathological tool payloads, retention, or native bugs — not
+   "unbounded context." (See §5.)
+
+---
+
+## 4. What was ruled out
+
+- **Warm-cache probe** — confirmed (Gemini + my read) as a *latent* unhandled-rejection path
+  (`createWarmService`'s `warm` has no try/catch around its `for await (provider.stream(...))` at
+  orchestrator.ts:1276; `fireWarm` drops the promise with `void` at warmer.ts:130). **But it has
+  never fired in production** — zero `cache-warming` activity in the journal, zero enabled
+  conversations in the `kv` table (53k rows, all conversation settings; warming is opt-in default
+  OFF). It is a latent risk worth fixing (the `unhandledRejection` guard from §1 catches its
+  rejection too), but it is **not** the cause of any observed crash.
+- **LSP** — exonerated (per handoff §7; caches bounded; disabled as a precaution, unrelated to
+  crashes).
+- **Idle/background leak** — refuted by telemetry (idle flat at 84 MB).
+- **`activeConversations` cap as a fix** — rejected. It's a workaround that cripples the
+  concurrent-agent product; the existing provider-concurrency limiter (3-4) already bounds
+  provider streams. The root cause is the missing error listener + missing guard.
+
+---
+
+## 5. Memory: what actually drives GB-scale (and what doesn't)
+
+The user's key insight: **300k tokens ≈ 1-2 MB** — raw context size is never GB-scale (capped at
+single-digit MB by the context window). So "unbounded context → GB crash" is numerically wrong.
+The only ways a turn reaches GB-scale:
+
+1. **A pathological tool payload** — a single tool result that's itself hundreds of MB (huge
+   file/log read). It enters `messages` unbounded and each step re-`JSON.stringify`s it (stream.ts:91)
+   **plus** the `reqSpan` captures a second copy (stream.ts:112) → ~2× transient per step.
+   **Tool-output bounding** directly prevents this (opencode caps at 50 KB / 2000 lines,
+   offloading full text to a managed file — `packages/opencode/src/tool/truncate.ts`).
+2. **O(N²) re-serialization across many steps**, only if GC can't keep up or there's retention.
+3. **A genuine retention leak** (message arrays / spans / closures held after the step or turn) —
+   accumulates per-turn over time (the 6.2 GB-over-2.5 h pattern). **Bounding context does NOT
+   fix this** — the retained reference must be found. (Not yet investigated; the span's
+   `bodyString` copy is written to the journal sink on disk, so it's likely transient, not
+   retained — but unverified.)
+
+**Dispatch vs opencode (context hygiene — NOT crash fixes):**
+| Aspect | Dispatch | opencode |
+|---|---|---|
+| Mid-turn compaction | refuses during active conv (orchestrator.ts:1325) | compacts between steps near context window (llm.ts:210) |
+| Tool result size in history | unbounded | bounded 50 KB / 2000 lines (truncate.ts) |
+| Step limit | `MAX_STEPS = 0` (unlimited) | configured per agent; `MAX_STEPS_PROMPT` disables tools at last step |
+| History sent to provider | full raw `messages` array | projected `toLLMMessages(...)` after compaction |
+
+These are general hygiene improvements, **not** crash fixes (the live crash is the SSH bug; the
+segfault is native). Tool-output bounding is the one with direct crash relevance (prevents
+pathological-payload spikes). Compaction/max-steps do not reduce RAM for context the agent
+genuinely needs.
+
+---
+
+## 6. Recovery behavior on restart (repair-and-seal, NOT resume)
+
+When the server crashes and systemd restarts it:
+- **`conversation-store/extension.ts:21-27`** — boot-sweep: lists all conversations with
+  `status: ["active"]` and sets them to `"idle"`.
+- **`store.ts:681`** — on load, `reconcileWithReport` repairs partial turns: synthesizes error
+  results for orphaned tool-calls, strips error-only trailing assistant messages, drops empty
+  messages. Emits a `reconcile.repair` span when repairs occur.
+- **`heartbeat/heartbeat.ts:299`** — sweeps stale "running" runs to "stopped".
+
+**Result:** the DB is always consistent after a restart (no broken conversations), but in-flight
+turns are **sealed, not resumed** — partial assistant output is preserved (reconciled), the
+conversation is marked idle, and the user must re-send. This is why the `uncaughtException` guard
+matters: without it, one SSH error kills the process → **every** in-flight turn (not just the one
+using SSH) is sealed and must re-send. With the guard, only the failing turn seals; the rest
+continue.
+
+---
+
+## 7. Data artifacts & quick-reference
+
+```bash
+# Live state
+systemctl status dispatch
+systemctl show dispatch -p MemoryMax -p MemoryHigh          # 24G / 20G (live, applied)
+curl http://localhost:24991/health                            # → {"ok":true}
+
+# Telemetry lives HERE (not journald):
+/var/log/dispatch/app.ndjson        # current process
+/var/log/dispatch/app.ndjson.1      # rotated (older crash windows rotated away — only .1 kept)
+# grep: rg 'memory:periodic|memory:gc' /var/log/dispatch/app.ndjson
+
+# Journal (crash stacks):
+journalctl -u dispatch --since '24 hours ago' | rg 'Main process exited|panic|Bun v'
+
+# The two investigation docs:
+notes/memory-leak-investigation-handoff.md   # prior (has the 3 errors in §3)
+notes/crash-investigation-findings.md         # THIS file (definitive)
+crash-review-report.md                        # independent Gemini review (found the SSH bug)
+
+# Suspect code:
+packages/ssh/src/pool.ts                      # buildConnection/doConnect — the crash
+packages/host-bin/src/main.ts                 # missing uncaughtException guard
+packages/kernel/src/runtime/run-turn.ts       # MAX_STEPS=0, streaming retry loop
+packages/openai-stream/src/stream.ts          # JSON.stringify whole body + span copy
+packages/session-orchestrator/src/orchestrator.ts  # warm probe (latent), activeConversations
+```
+
+## 8. Current fix scope
+
+**In scope now (root-cause, Dispatch code):**
+1. `packages/ssh/src/pool.ts` — permanent `'error'` listener on the pooled `ssh2.Client` + logging.
+2. `packages/host-bin/src/main.ts` — `uncaughtException` + `unhandledRejection` guards with rich
+   logging (message, stack, activeConversations, memory snapshot, timestamp).
+
+**Deferred (separate decisions):**
+- Tool-output bounding, mid-turn compaction, MAX_STEPS (general hygiene; not crash fixes).
+- Warm-probe try/catch (latent; the `unhandledRejection` guard covers it for now).
+- Port to Node (only if segfault recurs on 1.3.14 under real load).
+- Core-dump capture setup (operational; do before the next load test).
diff --git a/notes/memory-leak-investigation-handoff.md b/notes/memory-leak-investigation-handoff.md
new file mode 100644
index 0000000..3dc4ad0
--- /dev/null
+++ b/notes/memory-leak-investigation-handoff.md
@@ -0,0 +1,331 @@
+# Memory-Leak Investigation — Handoff for OpenCode Agent
+
+> **Purpose:** This document gives an OpenCode harness agent — running outside
+> the Dispatch system, with no prior context — everything needed to
+> independently investigate the memory leak that is crashing the production
+> Dispatch server. Read it top to bottom before touching anything.
+
+**Last updated:** 2026-06-28
+**Author of this handoff:** umans/umans-glm-5.2 (Dispatch agent)
+**Originating investigation:** `notes/server-crash-investigation.md` (commit `b83aa8d`)
+
+---
+
+## 0. TL;DR — what you are here to do
+
+The production Dispatch server leaks memory at **~2.5 GB/hour** and eventually
+hits a **Bun runtime segfault** (native crash, not a JS exception). A prior
+investigation already ruled out LSP as the cause. Your job is to **find where
+memory is being retained** and propose a fix. There are four undeployed commits
+on the `dev` branch that add memory telemetry and a cgroup circuit breaker —
+**those need to be built, installed, and observed first** to get the data that
+will localize the leak. Do not start code-diving blindly; deploy the telemetry,
+collect a crash cycle, then read the logs.
+
+---
+
+## 1. System Overview — what Dispatch is
+
+Dispatch is an **AI agent orchestration platform**: a backend that runs one or
+more AI "agents" through turns (each turn = a step loop of model calls +
+tool dispatch), plus a separate frontend that consumes the backend's typed
+contracts over HTTP + WebSocket.
+
+**Repo layout** — all under `/home/tradam/projects/dispatch/`:
+
+| Path | What | Git remote | Branch |
+|---|---|---|---|
+| `backend/` | The server (this codebase) | `[email protected]:realtradam/dispatch.git` | `dev` |
+| `frontend/` | The web UI (separate repo, same methodology) | `[email protected]:realtradam/frontend.git` | `dev` |
+| `worktrees/` | Per-task git worktrees for isolated feature branches | (varies) | (varies) |
+| `bin/` *(inside backend)* | Operational shell scripts (build, install, up, secrets, certs) | — | — |
+
+Both repos use **`dev` as the active development branch**. The backend is a
+**Bun + TypeScript** monorepo (project references via `tsc -b`, Biome for
+lint/format, Vitest for tests, `bun:sqlite` for storage, the `ai` /
+`@ai-sdk/*` packages for model providers). Architecture rules are in
+`backend/AGENTS.md` — **read it** before editing (especially the "kernel
+touches no I/O" and "no ambient state" rules).
+
+---
+
+## 2. Where things are installed (production)
+
+| Artifact | Path | Notes |
+|---|---|---|
+| Production server binary | `/usr/bin/dispatch-server` | Bun **standalone compiled** binary. Currently the OLD one (built Jun 27 21:40). Not you to install — see §6. |
+| CLI binary | `/usr/bin/dispatch` | `dispatch send/list/read/...` |
+| Frontend static files | `/usr/share/dispatch/web/` | Built by `bin/build` (Vite, `VITE_HTTP_PORT=24991 VITE_WS_PORT=24990`) |
+| Config / env file | `/etc/dispatch/env` | `EnvironmentFile` for systemd. **Root-readable only** (`Permission denied` for non-root). Source template in repo: `systemd/dispatch.env` (installed by `bin/setup-env`). |
+| Data directory | `/var/lib/dispatch/` | WorkingDirectory of the service. SQLite DBs live here: `dispatch.db`, `dispatch.db-shm`, `dispatch.db-wal`. |
+| Logs | the journal | `journalctl -u dispatch` — there are no log files on disk; everything goes to journald. |
+| systemd unit (live) | `/etc/systemd/system/dispatch.service` | Installed from the repo template `systemd/dispatch.service` by `bin/install`. |
+| systemd unit (repo template) | `backend/systemd/dispatch.service` | **Edit this one**, not the live file — `bin/install` copies it over. |
+| Install script | `backend/bin/install` | Builds + installs the binary, unit, env, frontend. Uses `sudo` on privileged lines only (the agent has no sudo — hand scripts to the user). |
+| Build script | `backend/bin/build` | `tsc --build` then `bun build --compile` → `dist/dispatch-server`. |
+| Memory-limits script | `backend/bin/apply-memory-limits.sh` | Applies `MemoryHigh`/`MemoryMax` to the live service. **User has NOT run it yet.** |
+
+---
+
+## 3. Production ports
+
+| Service | Port | Confirmed |
+|---|---|---|
+| HTTP API | **24991** | `BACKEND_PORT=24991` in `systemd/dispatch.env` (set by `bin/setup-env`). Live: `ss -tlnp` shows `dispatch-server` listening on `*:24991`. |
+| Surface WebSocket | **24990** | `SURFACE_WS_PORT`. Live: `ss -tlnp` shows `dispatch-server` listening on `*:24990`. |
+
+The **dev** stack (`bin/up`) uses different ports: **24203** (HTTP) + **24205** (WS) + **24204** (frontend Vite dev server). Don't confuse dev ports with prod ports.
+
+**Health check:** `curl http://localhost:24991/health` → `{"ok":true}`
+
+---
+
+## 4. How to access the running server
+
+```bash
+# Is it running?
+systemctl status dispatch
+
+# Live logs (follow)
+journalctl -u dispatch -f
+
+# Recent crashes / errors (last 2h)
+journalctl -u dispatch --since '2 hours ago' -n 200
+
+# Memory telemetry — ONLY visible once the new binary is deployed (see §6).
+# The EXACT log tags to grep (NOT "[mem-telemetry]" — that is the module name):
+journalctl -u dispatch -f | rg 'memory:periodic|memory:gc'
+
+# Restart (needs root — hand to the user)
+sudo systemctl restart dispatch
+```
+
+> **The service is named `dispatch`, NOT `dispatch-server`.** The binary is
+> `dispatch-server`; the systemd unit is `dispatch.service`. This mismatch
+> tripped up the first investigation — don't repeat it.
+
+The backend checkout at `/home/tradam/projects/dispatch/backend/` is on `dev`
+and contains the source. The running binary is compiled, so **editing source
+does nothing until you rebuild + install + restart** (§6).
+
+---
+
+## 5. The memory leak — what we know
+
+### 5.1 Symptom & crash signature
+
+- The server grows **~2.5 GB/hour** under load, eventually triggering a **Bun
+  runtime segfault** (native crash).
+- **Last crash:** Jun 28 00:12 JST.
+  ```
+  panic(main thread): Segmentation fault at address 0x0
+  oh no: Bun has crashed. This indicates a bug in Bun, not your code.
+  Main process exited, code=dumped, status=4/ILL
+  ```
+  RSS was **6.2 GB** at crash (over 2h31m uptime). A prior session hit
+  **25.7 GB over 18h**.
+- The crash is a **native segfault inside Bun's runtime** (NULL deref in the
+  allocator/GC), NOT a JS exception. The crash-time memory readout was itself
+  corrupted (`RSS: 0.02ZB` — impossible), and systemd saw SIGILL while Bun
+  reported SIGSEGV — both signs of **heap corruption under memory pressure**.
+- This is the **only** segfault in 5 days of logs. Earlier crashes (Jun 27
+  ~02:44–02:52) were `exit-code` failures = application-level LSP bugs, **now
+  fixed** (see §7).
+
+### 5.2 What it is NOT
+
+- **NOT LSP.** The Language Server Protocol extension was the original prime
+  suspect; it is exonerated. Its caches are bounded (`MAX_OPEN_DOCUMENTS = 50`,
+  `MAX_PUSH_DIAGNOSTICS = 100` — see `packages/lsp/src/client.ts:153` and
+  `diagnostics.ts:47`). 50 docs + 100 diag entries cannot explain gigabytes.
+- **NOT the conversation store.** `packages/conversation-store` is
+  **SQLite-backed** (all reads/writes go through a `StorageNamespace` to
+  `bun:sqlite`) — it does not hold conversations in an in-memory map.
+- **NOT `activeConversations`.** The session-orchestrator's
+  `activeConversations` is just a `Set<string>` of conversation IDs (tiny).
+
+### 5.3 Prime suspects (not yet confirmed — that's your job)
+
+1. **The AI-SDK streaming path** — the `async` generator returned by
+   `provider.stream(...)` (`@ai-sdk/*`, including the `openai-stream` package).
+   Streaming response buffers may be retained if the generator is not fully
+   drained or if the SDK holds internal buffers per request.
+2. **Per-turn message arrays** in `session-orchestrator` — the history +
+   `userMsg` + `providerMessages` arrays assembled before each
+   `provider.stream(...)` call. For long multi-step agent turns these can be
+   large; if references outlive the turn (closures, unresolved promises, event
+   listeners), they leak.
+3. **A Bun bug fixed in 1.3.14.** The crash was on Bun **v1.3.13**. Bun 1.3.14
+   has been built locally but not installed (§6.3) — deploying it may itself
+   resolve the segfault even if a leak remains.
+
+### 5.4 Key files to examine (the leak-suspect code path)
+
+| File | What's there | Why it matters |
+|---|---|---|
+| `packages/session-orchestrator/src/orchestrator.ts` | `runTurnDetached()` at **line 541** — kicks off a turn; the `activeConversations.add/delete` around it (556, 979) brackets the turn lifecycle. | Turn lifecycle: where memory should be acquired and released. If a turn's buffers aren't released here, it leaks per-turn. |
+| `packages/session-orchestrator/src/orchestrator.ts` | `for await (const event of provider.stream(messages, assembled.tools, providerOpts))` at **line 1276** (and a second one at **1430** for compaction/summary). | **The streaming loop** — the #1 suspect. This is where the AI-SDK async generator is consumed. If the generator is abandoned mid-stream (error, `break`, abort) or its internal buffers are retained, memory grows per turn. |
+| `packages/session-orchestrator/src/pure.ts` | The pure turn/step decision logic + the `MemorySample`/`memoryDelta`/`memorySampleAttributes` helpers used by telemetry. | The per-turn before/after sampling wraps the stream boundary (referenced by `mem-telemetry.ts`). |
+| `packages/kernel/src/runtime/run-turn.ts` | The **step loop** (`runTurn`) — one agent turn = N steps of (model call → tool dispatch → feed results back). | MAX_STEPS is disabled (`0 = unlimited`, commit `e8b4bf1`), so a single turn can run many steps. Many steps ⇒ many accumulated message arrays ⇒ more retention surface. |
+| `packages/kernel/src/runtime/dispatch.ts` | **Tool dispatch** (the `maxConcurrent`/`eager` tool-execution loop). | Tool results accumulate into the turn's message history. A rogue tool returning huge payloads could balloon arrays. |
+| `packages/host-bin/src/mem-telemetry.ts` | The periodic telemetry edge effect. | Read this to understand exactly what `memory:periodic` / `memory:gc` log entries contain (rss, heapUsed, heapTotal, external, arrayBuffers, activeConversations, reclaimedRssMB). |
+
+> **Architecture note (AGENTS.md):** the kernel (`packages/kernel`) touches NO
+> I/O and names no concrete feature. Decision logic is pure; effects are
+> injected at the edges (host-bin). When investigating, keep this layering in
+> mind — a leak "in the kernel" would actually be in a closure captured by an
+> edge that feeds the kernel, not in the kernel's own (stateless) turn loop.
+
+---
+
+## 6. What's been done (committed to `dev`, NOT yet deployed)
+
+The running binary is still the **old one** (built Jun 27 21:40, pre-telemetry,
+pre-Bun-1.3.14, pre-LSP-disable). Four commits on `dev` need building +
+installing before any of this takes effect. **Verify before you trust:** the
+live systemd still reports `MemoryMax=infinity` (confirmed via
+`systemctl show dispatch -p MemoryMax`).
+
+| Commit | What | Status |
+|---|---|---|
+| `d1de9ed` | `systemd/dispatch.service` gets `MemoryHigh=20G` + `MemoryMax=24G` circuit breaker. | Committed. **Live = infinity (not applied).** |
+| `b3b6eb4` | `bin/apply-memory-limits.sh` — applies the limits to the live service. | Committed. **User has NOT run it.** |
+| `1cd66da` | Memory telemetry: periodic `process.memoryUsage()` every 15s, per-turn before/after sampling, `activeConversations` tagging, `Bun.gc(true)` every 5 min. Files: `packages/host-bin/src/mem-telemetry.ts` + `packages/session-orchestrator/src/pure.ts`. | Committed. **NOT deployed.** |
+| `73ff84c` | **LSP disabled** as a precaution (`main.ts` import commented out + removed from `CORE_EXTENSIONS`; `transport-http` tolerates its absence). Also set telemetry interval 60s→15s. | Committed. **NOT deployed.** LSP stays disabled until the leak is understood. |
+| (bun upgrade) | Bun **1.3.13 → 1.3.14** via `bun upgrade`. New binary built at `dist/dispatch-server`. | Rebuilt. **NOT installed** to `/usr/bin/dispatch-server`. |
+
+### 6.1 The telemetry data you will collect (once deployed)
+
+The exact log tags to grep in journald are **`memory:periodic`** and
+**`memory:gc`** (the module is `mem-telemetry.ts`, but the log *messages* are
+those two strings — do not grep for `[mem-telemetry]`).
+
+- `memory:periodic` — every 15s: `rss`, `heapUsed`, `heapTotal`, `external`,
+  `arrayBuffers` (bytes), plus `activeConversations` (count). Correlate RSS
+  growth with active turns: if RSS climbs while `activeConversations` is 0,
+  the leak is background/idle (timers, watchers, retained closures); if it
+  climbs only during/after turns, it's the streaming/turn path.
+- `memory:gc` — every 5 min: runs `Bun.gc(true)` and logs RSS before/after +
+  `reclaimedRssMB` (`before - after`). **Interpretation:** a large positive
+  `reclaimedRssMB` = GC freed it (fragmentation, reclaimable — not a true
+  leak). Near-zero/negative `reclaimedRssMB` = memory is **LIVE** (retained
+  objects — a real leak). This single field distinguishes the two failure
+  modes; check it first when you get data.
+
+### 6.2 Deploy sequence (hand to the user — you have no sudo)
+
+The agent cannot run `sudo` or install to `/usr/bin`. Write a script with
+`sudo` on the privileged lines and hand it to the user (per the repo's sudo
+policy). The sequence, roughly:
+
+1. From `backend/`: `bun run typecheck && bun run test && bun run check` —
+   make sure `dev` is green (it should be; these commits are committed).
+2. `bin/build` — produces `dist/dispatch-server` (Bun 1.3.14, with telemetry +
+   LSP-disabled).
+3. `bin/install` (or a targeted script) — copies `dist/dispatch-server` →
+   `/usr/bin/dispatch-server`, the unit template → `/etc/systemd/system/`,
+   `systemd/dispatch.env` → `/etc/dispatch/env`. (Or run `bin/apply-memory-limits.sh`
+   separately if you only want the cgroup limits without a full reinstall.)
+4. `sudo systemctl daemon-reload && sudo systemctl restart dispatch`.
+5. Confirm: `systemctl show dispatch -p MemoryMax` should show `24G`, and
+   `journalctl -u dispatch -f | rg 'memory:periodic'` should show telemetry
+   within 15s.
+
+> ⚠️ **Warning:** deploying restarts the production server. The live server is
+> actively serving agent conversations. Coordinate with the user (ceb2 /
+> tradam) before restarting — don't just yank it. Also note the running server
+> at time of writing is PID 28956 (started Jun 28 08:51).
+
+---
+
+## 7. Background — the original crash investigation (already done)
+
+You do **not** need to redo this; it's recorded for context.
+
+- **Original report:** `notes/server-crash-investigation.md` (commit `b83aa8d`).
+- The first investigation's task description was **stale**: it claimed an
+  "uncommitted hot-fix that disables LSP" existed. It did not — the working
+  tree was clean and LSP was *enabled*. The developer then *chose* to disable
+  LSP (commit `73ff84c`) as a precaution, but the root cause was already
+  identified as the Bun segfault + leak, not LSP.
+- The **three original LSP suspects** (JSON-parse TypeError, `fs.watch` ENOENT,
+  unbounded cache leak) were all fixed in commits `05ff256` + `f9d1ca5` and
+  are correct/tested. LSP being disabled now is a precaution, not a fix for the
+  crash. Once the leak is understood, LSP can be re-enabled safely.
+- **Do not spend time re-investigating LSP.** It is exonerated.
+
+---
+
+## 8. What needs investigation (your tasks, in order)
+
+1. **Deploy the telemetry + cgroup limits** (§6) — you cannot localize the leak
+   without the `memory:periodic` / `memory:gc` data. Get a full crash cycle's
+   worth of logs (let it run until it either hits `MemoryMax=24G` and restarts,
+   or until the growth pattern is clear).
+2. **Read the telemetry first.** The `memory:gc` `reclaimedRssMB` field tells
+   you live-retained vs fragmentation. If near-zero after GC → real retained
+   leak; chase which subsystem holds it.
+3. **Correlate growth with `activeConversations`.** Idle-growth ⇒ timers /
+   watchers / retained closures / the LSP file-watcher (if any servers still
+   spawn — but LSP is disabled). Turn-bound growth ⇒ the streaming/turn path
+   (suspects in §5.4).
+4. **Examine the streaming loop** (`orchestrator.ts:1276`): is the
+   `provider.stream(...)` async generator fully drained on every path
+   (success, error, abort, `break`)? Does the AI SDK retain response buffers?
+   Consider adding a before/after `memoryUsage()` around a single turn to
+   measure per-turn delta directly.
+5. **Examine the message arrays** assembled before `provider.stream` — are
+   they scoped to the turn and released when the turn completes, or do
+   closures/promises/event-listeners keep them alive?
+6. **Test Bun 1.3.14 in isolation.** Since the upgrade is built but not
+   installed, see if a memory-stress repro under 1.3.13 vs 1.3.14 behaves
+   differently — the segfault may be a Bun allocator bug that 1.3.14 fixes.
+7. **Propose a fix** (do not implement without coordinating — report up to the
+   orchestrator). Likely candidates: ensure the streaming generator is always
+   fully consumed / explicitly closed on error; bound per-turn message
+   history; release references at `activeConversations.delete` time; or
+   confirm it's purely a Bun bug requiring the 1.3.14 deploy.
+
+---
+
+## 9. Quick-reference commands
+
+```bash
+# --- status & logs ---
+systemctl status dispatch
+systemctl show dispatch -p MemoryMax -p MemoryHigh       # expect 24G/20G after deploy
+journalctl -u dispatch -f                                  # live logs
+journalctl -u dispatch -f | rg 'memory:periodic|memory:gc' # telemetry (after deploy)
+journalctl -u dispatch --since '2 hours ago' -n 200       # recent history / crashes
+curl http://localhost:24991/health                        # → {"ok":true}
+ss -tlnp | rg dispatch                                    # ports 24991 + 24990
+
+# --- source (the leak path) ---
+cd /home/tradam/projects/dispatch/backend
+git log --oneline -n 8                                    # see the undeployed commits
+rg -n 'runTurnDetached|provider\.stream|for await' packages/session-orchestrator/src/orchestrator.ts
+
+# --- build & verify (NO sudo needed) ---
+bun run typecheck && bun run test && bun run check
+bin/build                                                 # → dist/dispatch-server
+
+# --- deploy (NEEDS sudo — hand a script to the user) ---
+# bin/install  (or: sudo cp dist/dispatch-server /usr/bin/dispatch-server)
+# bin/apply-memory-limits.sh   (applies MemoryMax to live service)
+# sudo systemctl daemon-reload && sudo systemctl restart dispatch
+```
+
+---
+
+## 10. Guardrails
+
+- **Do NOT edit implementation code without reporting to the orchestrator.**
+  This is an investigation handoff; propose fixes, don't apply them
+  unilaterally.
+- **You have no sudo.** Any step touching `/usr/bin`, `/etc/`, or
+  `systemctl restart` must be a script handed to the user with `sudo` on the
+  privileged lines.
+- **Do not re-enable LSP** until the leak is understood (it's disabled as a
+  precaution; re-enabling is a separate decision).
+- **Do not merge or push.** Work stays on `dev` locally.
+- **The service is `dispatch`, not `dispatch-server`.**