| Age | Commit message (Collapse) | Author |
|---|
|
|
|
uncaughtException/unhandledRejection guards
Root cause of the live production crash (exit-1 'Timed out while waiting for
handshake'): the pooled ssh2.Client had no permanent 'error' listener after
connect, so a post-connect ssh2 error escaped as an uncaught EventEmitter
'error' with no process-level guard. See notes/crash-investigation-findings.md §1.
- packages/ssh/src/pool.ts: attach a permanent 'error' listener to the pooled
client in buildConnection that sets state=error, logs (alias, message, level),
and does not throw; cleanup() no longer removes it.
- packages/host-bin/src/main.ts: add process.on('uncaughtException') (graceful
shutdown after logging) and process.on('unhandledRejection') (log + continue),
both logging message/stack, memory snapshot, activeConversations count, and
timestamp so the failure site is observable.
|
|
|
|
- Disable LSP extension (import + CORE_EXTENSIONS) due to crashes
- Make transport-http tolerate LSP being absent (optional getService)
- Remove lsp from transport-http dependsOn manifest
- Change memory telemetry sample interval from 60s to 15s
|
|
|
|
service
Hands the user a one-command script to apply the systemd MemoryMax/MemoryHigh
cgroup limits to the live dispatch.service without a full reinstall.
Patches in the real user (same as bin/install), daemon-reloads, and restarts.
|
|
The server has a memory leak (~2.5 GB/h) that eventually triggers a Bun
runtime segfault. These cgroup limits turn the uncontrolled crash into a
controlled OOM-kill → clean restart via Restart=on-failure.
MemoryHigh=20G (soft throttle) + MemoryMax=24G (hard cap).
Machine has 33.24 GB total; 24G leaves OS headroom.
|
|
|
|
background files
|
|
(document lifecycle) + leaked init promises
|
|
# Conflicts:
# packages/session-orchestrator/src/extension.ts
# packages/session-orchestrator/src/orchestrator.ts
|
|
When a non-vision model (e.g. GLM) calls consult_vision, the new Kimi
consultation tab now shows 'IMAGE - <question>' instead of the bare
question-derived title, making image-consultation tabs visually
distinguishable from normal conversation tabs.
- Add formatConsultationTitle(question) pure helper (pure.ts): prefixes
'IMAGE - ' and truncates the question to 80 chars (matching the
conversation store's TITLE_MAX) with an ellipsis.
- Add setConversationTitle dep to VisionHandoffDeps, wired in the
extension to the conversation store's setConversationTitle.
- Call it in consultVision BEFORE the turn starts so the title is
correct from the first moment (the store keeps a non-'Untitled'
title on first message append). Best-effort: a title-write failure
logs a warning but does not break the consultation.
- Tests: 4 pure + 2 service (title set + optional-dep graceful).
|
|
purge on compaction/close
|
|
host storage
|
|
blocks on acquire, re-emit "active" when slot granted
|
|
settings
|
|
|
|
auto-transcription
|
|
rejects temperature: 0)
The vision handoff hardcoded temperature: 0 for the transcription sub-call,
but the Moonshot/Kimi vision model only allows temperature: 1 (or omitted),
causing an HTTP 400 "invalid temperature: only 1 is allowed for this model"
that blocked the entire image analysis for non-vision models like GLM 5.2.
Fix: omit temperature entirely so each vision provider uses its own default —
the truly universal, provider-agnostic approach (different providers have
different temperature constraints).
|
|
overshoot from provider accounting lag
|
|
|
|
|
|
|
|
vision handoff
|
|
limits with oldest-agent-first scheduling
|
|
|
|
# Conflicts:
# packages/host-bin/package.json
# packages/host-bin/src/main.ts
# packages/session-orchestrator/src/orchestrator.ts
# packages/system-prompt/src/service.test.ts
# packages/system-prompt/src/service.ts
# packages/system-prompt/src/types.ts
# packages/transport-contract/package.json
# packages/transport-http/package.json
# packages/transport-http/src/app.test.ts
# packages/transport-http/src/app.ts
# packages/transport-http/src/extension.ts
# packages/transport-http/tsconfig.json
# tsconfig.json
|
|
|
|
backend, dispatch-web → frontend)
|
|
|
|
|
|
|
|
|
|
|
|
The script previously required sudo for the entire script (id -u check),
which meant bin/build ran as root and created root-owned dist/ files.
On the next build, the normal user couldn't overwrite them (EACCES).
Now the script runs without a sudo prefix: the build step runs as the
normal user (dist/ files are user-owned), and sudo is used only on the
specific lines that write to system directories (/usr/bin, /etc,
/usr/share) or call systemctl.
|
|
bin/build was compiling the binary directly from stale dist/*.js files
without first recompiling the TypeScript packages. Since package.json
main fields point to dist/index.js, source edits to .ts files were
silently lost in the compiled binary.
Now tsc --build runs first (composite project references rebuild all
packages in dependency order), then bun build --compile bundles the
fresh dist/ output.
|
|
|
|
Agents were being cut off mid-task at 50 steps. The MAX_STEPS=50
hardcoded limit was silently terminating turns while the model was
actively making tool calls, leaving conversations idle with a
dangling tool-result as the last chunk.
Setting MAX_STEPS to 0 disables the limit — the loop runs until the
model stops making tool calls naturally or the abort signal fires.
The max-steps code path is preserved for when MAX_STEPS > 0.
|
|
|
|
The test endpoint's runProbe() waited for the ssh2 stream's 'close' event,
which some SSH servers never emit for short-lived exec channels (the command
'true' exits instantly). This caused the promise to hang forever — the HTTP
response never returned, and the FE's Test spinner spun indefinitely.
Three fixes:
1. runProbe now resolves on the 'exit' event (not 'close') — the command has
finished and the exit code is available. 'close' is kept as a fallback.
Stream data/stderr are drained to prevent buffer deadlocks.
2. runProbe has a 15s timeout safety net — if the exec callback or 'exit'
event never fires (e.g. server requires a pty for exec), the probe
resolves false instead of hanging forever.
3. The entire test() method is wrapped in a 30s Promise.race timeout —
even if pool.acquire() or pool.drop() hangs, the endpoint ALWAYS
responds with { ok, error? }.
The probe is fully non-interactive (no blocking prompts). tsc EXIT 0,
biome clean, 1756 tests pass.
|
|
Two improvements to the SSH support feature:
1. KNOWN_HOSTS DISCOVERY (packages/ssh):
Computers are now auto-discovered from ~/.ssh/known_hosts (every hostname
you've ever connected to) in ADDITION to ~/.ssh/config (explicit Host
aliases). Config entries take precedence (full params); known_hosts entries
get defaulted params (User=defaultUser, IdentityFile=null→pool probes
default keys, Port from [host]:port or 22, knownHost=true). Zero-config —
no ~/.ssh/config file needed; hosts just appear.
Reject list: dispatch.toml [ssh].reject = [...] (glob patterns like
github.com, *.ts.net) filters noise from the catalog. Read from both
the global ~/.config/dispatch/dispatch.toml and the project dispatch.toml.
Parsed with Bun.TOML.parse (zero deps). Only filters discovery (catalog);
specific lookups (getComputer/getStatus/test/connect) ignore the reject
list (it's a visibility filter, not access control).
New pure functions: parseKnownHosts(), isRejected(), globMatch().
+26 tests. tsc EXIT 0, biome clean, 1756 tests pass.
2. REMOTE SYSTEM-PROMPT AWARENESS (packages/system-prompt):
When a conversation has a computerId set (remote turn), the system prompt
now resolves system:os, system:hostname, git:branch/git:status, and
file: reads against the REMOTE machine — not the local host. Previously
the prompt always said 'Arch Linux (WSL)' + local hostname even when the
agent was connected to a remote Artix Linux machine.
The ResolverAdapters' hostname()/platform() are now async (so a remote
adapter can run 'hostname'/'uname -s' over SSH). The system-prompt
extension builds remote adapters from the ExecBackend (readFile→SFTP,
spawn→SSH exec). Cache invalidation now checks computerId (switching
computers rebuilds the prompt). The compaction path also threads
computerId. @dispatch/system-prompt now depends on @dispatch/exec-backend.
|
|
cross-cutting verified
FE confirmed whole-tree green (typecheck 0/0, 795/795 tests, biome clean, build
OK, git clean). All three handoffs GREEN with no integration gaps:
- provider-retry: yellow alert-warning bubble renders w/ countdown.
- SSH #1 wire types: defaultComputerId + Computer/ComputerEntry resolve.
- SSH #2 computer API: full src/features/computer/ feature wired + typecheck-clean.
Cross-cutting verified: provider-retry is WS-stream (TranscriptState.providerRetry
→ ChatView), computer is HTTP-only (AppStore.computerId → ComputerField sidebar) —
disjoint state/channels/regions/mount-keys; no collision. SSH support + provider-
retry integration is complete and validated end-to-end on both repos.
|
|
Brings dev's retry-with-backoff (the transient `provider-retry` AgentEvent the
web frontend consumes) + the LSP-dead-server per-edit-hang fix into the SSH
feature branch, alongside the SSH waves 0-5c.
All code files auto-merged cleanly (run-turn.ts, orchestrator.ts, runtime.ts,
wire/index.ts, tool-edit-file/extension.ts, run-turn.test.ts — both computerId
threading and retry-with-backoff coexist). Only tasks.md conflicted (status
section — orchestrator-resolved; both feature sections kept).
Verified post-merge: tsc -b EXIT 0, biome clean (391 files), 1730 vitest pass
+6 sshd-integration skipped (was 1690; +40 from dev's retry/LSP tests).
Wire dist rebuilt so the FE can re-sync the pinned @dispatch/wire dep and pick
up BOTH provider-retry AND the SSH Computer/defaultComputerId types.
No merge or push (into dev or otherwise).
|
|
|
|
The LSP diagnostics path hung up to 60s per edit whenever a configured Ruby
language server was dead or slow (the reported Steep langserver case): a
killed/crashed server was never detected (stayed "connected" forever), servers
were queried sequentially with a 60s budget each, and a corrupted-but-alive
server (Steep's ~3h phantom-SyntaxError drift) had no recovery.
Four fixes, all in packages/lsp/ (the tool-edit-file call site lowered to 10s):
1. Dead-process detection: SpawnedProcess.onExit (Bun proc.exited) + stdout-end
defence flip the client to error, dispose the rpc, kill the proc. The manager
re-spawns a fresh server after the 30s backoff. Dead servers are now skipped
(0s) instead of polled for 60s.
2. Concurrent fan-out + 10s hard cap: new aggregateDiagnostics queries all
matching servers at once, each capped at 10s. A non-responder is skipped
with "LSP took too long (>10s), skipped — raise this to the user" instead of
blocking the fast server's results. Replaces the vague "unusually long"
warning (now structurally impossible: slow is always false).
3. Corruption self-heal: a detector flags a server re-emitting identical
non-empty diagnostics despite the file changing; after 5 repeats the client
is marked broken and re-spawned. Clean files never trip it. (Acknowledged
false-positive risk on persistent unfixed errors; CLI type-check gate stays
authoritative.)
4. sendRequest timeout: hover/definition/references cap at 10s so they can't
hang the turn against a dead server; the initialize handshake keeps its 45s
race.
Verification: typecheck clean; 1573 tests pass (96 files), +15 new LSP tests
(86 in packages/lsp); biome clean. No kernel/contract changes; onExit is
internal to packages/lsp.
|
|
This was the OLD orchestrator manual (references the retired `opencode run`
CLI + `opencode-go/mimo-v2.5-pro`, MVP-era content). The current manual lives
at root ORCHESTRATOR.md (references the `dispatch` CLI + umans/umans-glm-5.2).
Unrelated housekeeping; split from the retry feature commit.
|
|
When the upstream LLM API returns a retryable error (HTTP 429 / 5xx
"overloaded"), the kernel now retries provider.stream() with a stepped
backoff, visibly, until the 8h cumulative-sleep budget is exhausted — then
emits the final error and seals the turn. Retries fire only when no content
was emitted yet this step (safety invariant: never duplicate partial output).
- wire: new transient TurnProviderRetryEvent AgentEvent variant (emitted
before each sleep; not persisted to model history).
- kernel contracts: RetryStrategy (pure delayFor + injected sleep) + optional
retry? on RunTurnInput (omit = no retry, backward-compatible).
- kernel run-turn: retry loop in executeStep; providerRetryEvent constructor.
Kernel imports no timer (sleep injected).
- session-orchestrator: concrete schedule (5s..30m, repeat 30m, 8h budget) +
abortable setTimeout sleep, wired into RunTurnInput.retry.
tsc -b EXIT 0; biome clean; 1574 vitest pass (+16 new: 11 kernel retry tests
with injected fake sleep + pure delayFor, zero @dispatch/* mocks; 5 schedule
tests). Transports unchanged (transport-ws forwards AgentEvent verbatim in
chat.delta; transport-http is generic JSON.stringify).
Plan: notes/retry-with-backoff-plan.md. tasks.md updated with milestone +
optional CLI-renderer roadmap follow-up.
|
|
barrel
Wave 5c (final wiring) of transparent SSH support.
- host-bin: register exec-backend + ssh in CORE_EXTENSIONS (exec-backend before
the tool extensions that dependsOn it; ssh after, provides the remote-backend
factory + ComputerService at boot). +@dispatch/exec-backend/@dispatch/ssh deps +
tsconfig refs.
- transport-http: CR-5 — re-export computerServiceHandle + ComputerService type
from the package barrel (src/index.ts), mirroring lsp/mcp handles, so ssh imports
the typed symbol cleanly (no more dist/seam.js subpath workaround).
- orchestrator: added the @dispatch/exec-backend dep the host-bin agent missed +
bun install.
LIVE-VERIFIED: bun packages/host-bin/src/main.ts boots clean ('Dispatch booted',
no disabled extensions) — exec-backend + ssh + all tool extensions load together.
Verified: tsc -b EXIT 0, biome clean, 1690 vitest pass (+6 sshd-integration skipped).
DEFERRED (CR-6): listComputers usageCount stays 0 until a conversation-store
count-by-alias helper is added (non-blocking).
Refs: notes/ssh-support-plan.md. No merge or push.
|
|
Wave 5b of transparent SSH support. NEW standard extension @dispatch/ssh makes
remote execution actually work over SSH, transparently. ssh2 verified to run under
Bun (load-bearing decision #1 confirmed: connects to local sshd :22 + execs).
- config.ts: ~/.ssh/config reader via ssh-config -> Computer[]/ComputerEntry[]
(read-only discovery; resolves hostName/port/user/identityFile/knownHost).
- hostkey.ts: known_hosts auto-trust-and-pin (present->verify/reject-on-mismatch,
absent->accept+append; the accept-new analog).
- errors.ts: pure ssh2/SFTP -> node:fs-style .code error mapping (so tools'
existing ENOENT branches work unchanged).
- pool.ts: SshConnectionPool (per-alias ssh2.Client, lazy connect, keep-alive,
idle reap ~15m); key-only auth from ~/.ssh (config IdentityFile or default
id_ed25519/id_rsa); no agent-forwarding, no PTY.
- backend.ts: SshExecBackend implements ExecBackend (spawn via client.exec with
shell-quoted cwd; fs via SFTP).
- service.ts + extension.ts: activate provides BOTH handles the other units
consume — remoteExecBackendFactoryHandle (exec-backend: computerId->SshExecBackend)
AND computerServiceHandle (transport-http: listComputers/getComputer/getStatus/test).
- orchestrator: added packages/ssh to root tsconfig.json refs + bun install.
Tests: 45 pass + 6 sshd-integration skipped (it.skipIf(!process.env.SSH_TEST_HOST)).
Verified: tsc -b EXIT 0, biome clean, 1690 vitest pass (was 1641, +49).
CRs for wave 5c: host-bin registration; CR-5 transport-http barrel re-export;
CR-6 usageCount wiring (deferred-ok, defaults to 0).
Refs: notes/ssh-support-plan.md (decisions §0.5/§13). No merge or push.
|
|
exec-backend declares remoteExecBackendFactoryHandle (a consumer-defined
ServiceHandle<(computerId) => ExecBackend>) that the ssh package will provide
(standard→core layering). The resolver's computerId-set branch now lazy-looks-up
this factory (at tool-execute time, runtime) and calls it; if ssh isn't loaded,
getService throws → a clear 'SSH remote execution is not configured' error. The
computerId-undefined (local) branch is byte-identical to before.
This is the seam wave 5b (the ssh package) plugs into. +tests for both branches.
Verified: tsc -b EXIT 0, biome clean. No merge or push.
|
