diff options
| -rw-r--r-- | CS_ARTIX_DEPLOY.md | 132 | ||||
| -rw-r--r-- | Dockerfile | 22 | ||||
| -rw-r--r-- | Dockerfile.dev | 22 | ||||
| -rw-r--r-- | UPSTREAM_CS_FUZZY_BUG.md | 111 | ||||
| -rwxr-xr-x | bin/install-pkg | 6 | ||||
| -rw-r--r-- | docker/cs/fuzzy-distance.patch | 159 | ||||
| -rw-r--r-- | docker/cs/luau-declarations.patch | 32 | ||||
| -rw-r--r-- | packages/api/src/agent-manager.ts | 18 | ||||
| -rw-r--r-- | packages/api/tests/agent-manager.test.ts | 29 | ||||
| -rw-r--r-- | packages/api/tests/routes.test.ts | 8 | ||||
| -rw-r--r-- | packages/core/src/index.ts | 1 | ||||
| -rw-r--r-- | packages/core/src/tools/search-code.ts | 362 | ||||
| -rw-r--r-- | packages/core/src/tools/summon.ts | 2 | ||||
| -rw-r--r-- | packages/core/tests/agents/loader.test.ts | 2 | ||||
| -rw-r--r-- | packages/core/tests/tools/search-code.test.ts | 511 | ||||
| -rw-r--r-- | packages/frontend/src/lib/components/ToolPermissions.svelte | 5 | ||||
| -rw-r--r-- | packages/frontend/src/lib/settings.svelte.ts | 2 | ||||
| -rw-r--r-- | packaging/PKGBUILD | 56 |
18 files changed, 1474 insertions, 6 deletions
diff --git a/CS_ARTIX_DEPLOY.md b/CS_ARTIX_DEPLOY.md new file mode 100644 index 0000000..a942ef8 --- /dev/null +++ b/CS_ARTIX_DEPLOY.md @@ -0,0 +1,132 @@ +# Deploying the `cs` / `search_code` binary to the Artix (s6) cyberdeck + +## TL;DR + +The `search_code` agent tool shells out to a `cs` (code spelunker) binary. This +feature provisions that binary on **two** deployment paths automatically: + +| Path | Mechanism | `cs` ends up at | +| --- | --- | --- | +| `bin/up` | Docker (`Dockerfile` / `Dockerfile.dev`, `cs-builder` stage) | `/usr/local/bin/cs` | +| `bin/service install` | native Arch package `code-search` (built by `packaging/PKGBUILD`, installed by `bin/install-pkg`) | `/usr/bin/cs` | + +There is a **third** path — the Artix cyberdeck box — that is deployed by a +personal script living **outside this repo** +(`~/projects/cyberdeck/sync-dispatch.sh`). That script has now been edited (the +5 small additions below) so the Artix box also gets the patched `cs`. Before the +edit, `search_code` on the cyberdeck returned its graceful +`Error: search_code requires the 'cs' binary ...` message on every call. + +This file documents that follow-up (now applied). It is committed so the change +isn't forgotten and can be re-derived if the cyberdeck script is ever reset; the +actual edit lives in the cyberdeck repo, not here. + +--- + +## Why this is needed + +`packaging/PKGBUILD` now builds a `code-search` split package (a patched, +statically-linked `cs` pinned to upstream commit +`697e0bf194bbc7a4a877e5170c70618989fc92e7`, tag `v3.1.0`, plus two patches: +`docker/cs/luau-declarations.patch` for Roblox `.luau` declaration support and +`docker/cs/fuzzy-distance.patch` for correct fuzzy edit-distance matching). It +installs `cs` to `/usr/bin/cs`. + +`code-search` is a plain static binary with **no init-system coupling**, so it +installs and runs identically on Artix (Arch-based, `pacman`/`x86_64`). The only +gap is that `sync-dispatch.sh` — which pushes packages to the Artix box and +`pacman -U`s them — has a hardcoded two-package list (`dispatch` + `dispatch-s6`) +and does not yet include `code-search`. + +> Note: `sync-dispatch.sh` builds and pushes packages from the **main** dispatch +> checkout (`/home/tradam/projects/dispatch/packaging`), so this edit only +> becomes meaningful **after this feature branch is merged to `dev`** and that +> checkout rebuilds packages (`bin/build-pkg` / `sync-dispatch.sh --build`). + +--- + +## The edit applied to `~/projects/cyberdeck/sync-dispatch.sh` + +Five small additions (the four below plus mirroring `PKG_CS` into the generated +remote-script preamble alongside `PKG_DISPATCH` / `PKG_S6`). This edit has been +applied. To deploy, run `sync-dispatch.sh --build` (the `--build` flag rebuilds +the packages first, producing the new `code-search-*.pkg.tar.zst` that now +carries both the Luau and fuzzy patches). + +### 1. Declare the package name (next to `PKG_DISPATCH` / `PKG_S6`) + +```sh +PKG_DISPATCH="dispatch-0.0.1-1-x86_64.pkg.tar.zst" +PKG_S6="dispatch-s6-0.0.1-1-x86_64.pkg.tar.zst" +PKG_CS="code-search-0.0.1-1-x86_64.pkg.tar.zst" # <-- add +``` + +### 2. Add it to the "package exists" pre-check loop + +```sh +for pkg in "$PKG_DISPATCH" "$PKG_S6" "$PKG_CS"; do # <-- add "$PKG_CS" + if [ ! -f "${PKG_DIR}/${pkg}" ]; then + echo "ERROR: ${PKG_DIR}/${pkg} not found. Run with --build or 'bin/build-pkg' first." >&2 + exit 1 + fi +done +``` + +### 3. Add it to the `scp` upload + +```sh +scp -q "${PKG_DIR}/${PKG_DISPATCH}" "${PKG_DIR}/${PKG_S6}" "${PKG_DIR}/${PKG_CS}" "${TARGET}:/tmp/" +# ^^^^^^^^^^^^^^^^^^^^^^^ add +``` + +### 4. Add it to the remote `pacman -U` and cleanup `rm` + +Inside the remote script heredoc: + +```sh +pacman -U --noconfirm "/tmp/$PKG_DISPATCH" "/tmp/$PKG_S6" "/tmp/$PKG_CS" +rm -f "/tmp/$PKG_DISPATCH" "/tmp/$PKG_S6" "/tmp/$PKG_CS" +``` + +> The remote script is generated inside `sync-dispatch.sh` and references +> `$PKG_CS` via the same variable-expansion mechanism already used for +> `$PKG_DISPATCH` / `$PKG_S6`. Make sure `PKG_CS` is exported/substituted into +> the remote script the same way those two are (search the script for every +> place `PKG_S6` appears and mirror it for `PKG_CS`). + +No s6 service changes are needed — `code-search` ships only a binary, not a +service, so the existing `s6 repository sync` / `s6 set enable` dance is +unaffected. + +--- + +## Verifying on the Artix box after sync + +```sh +cs --version # -> cs version 3.1.0 +which cs # -> /usr/bin/cs +pacman -Q code-search # -> code-search 0.0.1-1 +``` + +Then, in a Dispatch tab with the `search_code` permission enabled, run a search; +it should return ranked results instead of the "cs binary not found" error. + +For a `.luau` sanity check (confirms the Luau patch is present), search a Roblox +project with `only: "declarations"` — `function` / `type` / `export type` lines +should be detected. + +For a fuzzy sanity check (confirms the fuzzy patch is present), a mid-word +deletion should match, e.g. `cs -- 'computSlipAngle~1'` finds `computeSlipAngle` +(returns empty on an unpatched cs). + +--- + +## If you ever decouple `cs` from this repo + +`code-search` is intentionally a standalone package (own name, own +`/usr/bin/cs`, upstream MIT license shipped). If `cs` later graduates to its own +AUR/repo package, the cleaner end state is to drop `package_code-search()` from +`packaging/PKGBUILD` and instead declare a `depends=('code-search')` (or the AUR +name) on the `dispatch` package — but as of this writing **no official or AUR +package for boyter/cs exists** (the AUR `cs` is an unrelated `ls`-with-icons +tool), so building it here is the correct approach. @@ -1,6 +1,25 @@ # Production Dockerfile — multi-stage build for the API server only # Frontend deploys separately (e.g., Cloudflare Pages) +# --- cs (code spelunker) builder --- +# Builds a patched, statically-linked `cs` binary for the search_code tool. +# Pinned to the v3.1.0 commit for reproducibility; the patch adds Luau +# declaration support and corrects fuzzy edit-distance matching (see +# docker/cs/luau-declarations.patch and docker/cs/fuzzy-distance.patch). cs vendors its +# dependencies, so the `go build` step is offline after the clone. +FROM golang:1.25-bookworm AS cs-builder +ARG CS_COMMIT=697e0bf194bbc7a4a877e5170c70618989fc92e7 +WORKDIR /build +COPY docker/cs/luau-declarations.patch /tmp/luau-declarations.patch +COPY docker/cs/fuzzy-distance.patch /tmp/fuzzy-distance.patch +RUN git clone https://github.com/boyter/cs.git src \ + && cd src \ + && git checkout "${CS_COMMIT}" \ + && git apply /tmp/luau-declarations.patch \ + && git apply /tmp/fuzzy-distance.patch \ + && CGO_ENABLED=0 go build -mod=vendor -ldflags="-s -w" -o /usr/local/bin/cs . \ + && /usr/local/bin/cs --version + FROM oven/bun:1 AS builder WORKDIR /app @@ -26,6 +45,9 @@ COPY --from=builder /app/node_modules ./node_modules COPY --from=builder /app/packages/core ./packages/core COPY --from=builder /app/packages/api ./packages/api +# Bundle the patched `cs` code-search binary for the search_code tool +COPY --from=cs-builder /usr/local/bin/cs /usr/local/bin/cs + # Create workspace directory for file tools RUN mkdir -p workspace diff --git a/Dockerfile.dev b/Dockerfile.dev index d239854..9614564 100644 --- a/Dockerfile.dev +++ b/Dockerfile.dev @@ -1,3 +1,22 @@ +# --- cs (code spelunker) builder --- +# Builds a patched, statically-linked `cs` binary for the search_code tool. +# Pinned to the v3.1.0 commit for reproducibility; the patch adds Luau +# declaration support and corrects fuzzy edit-distance matching (see +# docker/cs/luau-declarations.patch and docker/cs/fuzzy-distance.patch). cs vendors its +# dependencies, so the `go build` step is offline after the clone. +FROM golang:1.25-bookworm AS cs-builder +ARG CS_COMMIT=697e0bf194bbc7a4a877e5170c70618989fc92e7 +WORKDIR /build +COPY docker/cs/luau-declarations.patch /tmp/luau-declarations.patch +COPY docker/cs/fuzzy-distance.patch /tmp/fuzzy-distance.patch +RUN git clone https://github.com/boyter/cs.git src \ + && cd src \ + && git checkout "${CS_COMMIT}" \ + && git apply /tmp/luau-declarations.patch \ + && git apply /tmp/fuzzy-distance.patch \ + && CGO_ENABLED=0 go build -mod=vendor -ldflags="-s -w" -o /usr/local/bin/cs . \ + && /usr/local/bin/cs --version + FROM oven/bun:1 WORKDIR /app @@ -42,6 +61,9 @@ COPY packages/frontend/package.json packages/frontend/package.json # Install dependencies (cached unless package files change) RUN bun install +# Bundle the patched `cs` code-search binary for the search_code tool +COPY --from=cs-builder /usr/local/bin/cs /usr/local/bin/cs + # Source code is volume-mounted at runtime, overriding this copy COPY . . diff --git a/UPSTREAM_CS_FUZZY_BUG.md b/UPSTREAM_CS_FUZZY_BUG.md new file mode 100644 index 0000000..a177be3 --- /dev/null +++ b/UPSTREAM_CS_FUZZY_BUG.md @@ -0,0 +1,111 @@ +# Upstream bug: `cs` fuzzy search only matches substitutions, not insertions/deletions + +> Drafted for a potential PR/issue to [boyter/cs](https://github.com/boyter/cs). +> Dispatch ships a local fix as `docker/cs/fuzzy-distance.patch` (applied to the +> pinned cs build); this document is the upstream-facing writeup. + +## Summary + +`cs`'s fuzzy operator `term~N` is documented as *"fuzzy match within 1 or 2 +distance"* (`cs --help`, README), i.e. Levenshtein edit distance. In practice it +only ever matches **substitutions**: a mid-word **insertion** or **deletion** +that is genuinely edit distance 1 does **not** match. + +- `computSlipAngle~1` does **not** find `computeSlipAngle` (one dropped `e`) — edit distance 1. +- `houss~1` finds `house` (works — it's a substitution), but `hose~1` does **not** find `house` (one inserted `u`) — edit distance 1. + +Affected version: `cs version 3.1.0` (observed at commit `697e0bf`). The defect +is not reported in the issue tracker (checked all 51 open/closed issues + PRs as +of this writing). + +## Root cause + +`pkg/search/executor.go` implements fuzzy matching by scanning only +**same-length** windows of the content. For a term of length `L` it compares +every `L`-character substring against the term: + +```go +// fuzzyContains checks if any same-length substring of content matches +// the term within the given edit distance (substitution-based matching). +func fuzzyContains(content, term string, maxDist, termLen int) bool { + contentLen := len(content) + if contentLen == 0 || termLen == 0 || termLen > contentLen { + return false + } + for i := 0; i <= contentLen-termLen; i++ { + window := content[i : i+termLen] // <-- always exactly termLen long + if levenshtein(window, term) <= maxDist { + return true + } + } + return false +} +``` + +`fuzzyFind` (which produces match locations) has the same structure. Because +every candidate `window` is exactly `termLen` characters, the only edit the +comparison can ever observe is a **substitution**. An insertion or deletion +changes the string length by one, so the matching substring of the content is +`termLen ± 1` long and is never formed — hence never compared. + +This also makes one of the existing tests assert buggy behavior: +`{"Fuzzy Distance 2", "hovze~2", ...}` expects to match only `house`'s 5-char +window, but `hovze` is also within distance 2 of the 4-char window `" ove"` +elsewhere in the corpus — a match the same-length scan structurally cannot see. + +## Fix + +Scan windows of **every plausible length** in +`[termLen - maxDist, termLen + maxDist]` (clamped to ≥ 1) at each offset, and +keep the best (lowest-distance) window. A Levenshtein distance of `d` requires +the two strings' lengths to differ by at most `d` (each insert/delete shifts +length by one), so this range is exactly the set of window lengths that can be +within `maxDist`. `fuzzyFind` records the best window at each start and then +advances past it, so a single logical match doesn't produce a swarm of +overlapping locations. + +See `docker/cs/fuzzy-distance.patch` for the full diff. Shape: + +```go +func fuzzyWindowBounds(termLen, maxDist int) (int, int) { + minLen := termLen - maxDist + if minLen < 1 { + minLen = 1 + } + return minLen, termLen + maxDist +} + +// bestFuzzyMatchAt returns the length of the lowest-distance window starting at +// i that is within maxDist of term, or -1. Ties prefer the length closest to +// termLen. +func bestFuzzyMatchAt(content, term string, i, maxDist, minLen, maxLen int) int { ... } +``` + +## Complexity note + +Worst-case work per offset goes from one `levenshtein` call to `2*maxDist + 1` +calls. Since `cs` only supports `~1` and `~2`, that's a small constant factor +(≤ 5×) and only on fuzzy queries, which are rare relative to keyword/regex +searches. No measurable impact in practice. + +## Verification + +The change keeps `go test ./pkg/search/... ./pkg/ranker/... ./...` green (with +the one buggy assertion corrected) and adds explicit mid-word insertion/deletion +cases: + +```go +{"Fuzzy Distance 2", "houze~2", false, []string{"src/main/file1.go"}}, +{"Fuzzy Distance 1 deletion", "hose~1", false, []string{"src/main/file1.go"}}, // hose -> house (insert 'u') +{"Fuzzy Distance 1 insertion", "houxse~1", false, []string{"src/main/file1.go"}}, // houxse -> house (delete 'x') +``` + +End-to-end against a real `.luau` codebase, all four of these now match (they +returned empty before): + +``` +computSlipAngle~1 -> computeSlipAngle (CarPhysics.luau) +LauncTuning~1 -> LaunchTuning (LaunchController.luau) +LaunchTunin~1 -> LaunchTuning (LaunchTuning.luau) # already worked (suffix) +TireFrictio~1 -> TireFriction (CarPhysics.luau) # already worked (suffix) +``` diff --git a/bin/install-pkg b/bin/install-pkg index 72e218b..9665847 100755 --- a/bin/install-pkg +++ b/bin/install-pkg @@ -5,7 +5,7 @@ # Pass package names (without version) to install a custom set. # # Usage: -# bin/install-pkg # dispatch + dispatch-systemd +# bin/install-pkg # dispatch + dispatch-systemd + code-search # bin/install-pkg dispatch dispatch-s6 # dispatch + dispatch-s6 # bin/install-pkg dispatch dispatch-electron # dispatch + electron wrapper # bin/install-pkg --all # every freshest pkg found @@ -35,9 +35,9 @@ declare -a names declare -a paths if [ $# -eq 0 ]; then - names=(dispatch dispatch-systemd) + names=(dispatch dispatch-systemd code-search) elif [ "${1:-}" = "--all" ]; then - names=(dispatch dispatch-systemd dispatch-s6 dispatch-electron) + names=(dispatch dispatch-systemd dispatch-s6 dispatch-electron code-search) else names=("$@") fi diff --git a/docker/cs/fuzzy-distance.patch b/docker/cs/fuzzy-distance.patch new file mode 100644 index 0000000..e432986 --- /dev/null +++ b/docker/cs/fuzzy-distance.patch @@ -0,0 +1,159 @@ +Fix cs fuzzy matching to honour true Levenshtein edit distance (insertions and +deletions), not just same-length substitutions. + +Upstream cs (v3.1.0) implements `term~N` fuzzy search by scanning only +same-length windows of the content: for a term of length L it compares every +L-character substring and keeps those within N edits. Because every candidate +window is exactly L long, the only edits it can ever observe are substitutions. +A mid-word insertion or deletion — e.g. `computSlipAngle~1` for the real symbol +`computeSlipAngle` (a dropped 'e'), or `houss~1` vs `house` — changes the length +by one and so is never matched, even though it is edit distance 1. This +contradicts cs's own documentation ("fuzzy match within 1 or 2 distance"). + +The fix scans windows of every plausible length in +[termLen-maxDist, termLen+maxDist] (clamped at 1) at each offset and keeps the +best (lowest-distance) match, so insertions and deletions match too. fuzzyFind +records the best window per start and advances past it to avoid emitting a swarm +of overlapping locations for one logical match. + +Purely localised to the two fuzzy helpers in pkg/search/executor.go (plus a +test update: the pre-existing `hovze~2` case asserted the old substitution-only +behaviour, where the distance-2 term coincidentally failed to match a shorter +window that is in fact within distance 2; it is replaced with an unambiguous +distance-1 case and new mid-word insertion/deletion cases). Passes cs's own +pkg/search + pkg/ranker test suites. Applied during the Docker build and the +native package build via `git apply` against the pinned v3.1.0 checkout (see +Dockerfile / Dockerfile.dev / packaging/PKGBUILD). Candidate for upstreaming to +boyter/cs (the defect is unreported there). + +diff --git a/pkg/search/executor.go b/pkg/search/executor.go +index 175d458..1c422d2 100644 +--- a/pkg/search/executor.go ++++ b/pkg/search/executor.go +@@ -632,17 +632,67 @@ func min3(a, b, c int) int { + return c + } + +-// fuzzyContains checks if any same-length substring of content matches +-// the term within the given edit distance (substitution-based matching). ++// fuzzyWindowBounds returns the inclusive range of substring lengths that ++// could be within maxDist edits of a term of length termLen. A Levenshtein ++// distance of d can only be achieved between strings whose lengths differ by ++// at most d (each insertion or deletion changes the length by one), so any ++// candidate window must have a length in [termLen-maxDist, termLen+maxDist]. ++// The lower bound is clamped to 1 so we never form a zero-length window. ++func fuzzyWindowBounds(termLen, maxDist int) (int, int) { ++ minLen := termLen - maxDist ++ if minLen < 1 { ++ minLen = 1 ++ } ++ return minLen, termLen + maxDist ++} ++ ++// bestFuzzyMatchAt returns the length of the best (lowest-distance) substring ++// of content starting at offset i that is within maxDist edits of term, or -1 ++// if none is. Windows of every plausible length are tried (see ++// fuzzyWindowBounds) so insertions and deletions match, not just ++// substitutions — e.g. "computSlipAngle" (a dropped 'e') matches ++// "computeSlipAngle" at distance 1. Ties prefer the length closest to termLen. ++func bestFuzzyMatchAt(content, term string, i, maxDist, minLen, maxLen int) int { ++ contentLen := len(content) ++ bestLen := -1 ++ bestDist := maxDist + 1 ++ bestDelta := 0 ++ for wl := minLen; wl <= maxLen; wl++ { ++ if i+wl > contentLen { ++ break ++ } ++ d := levenshtein(content[i:i+wl], term) ++ if d > maxDist { ++ continue ++ } ++ delta := wl - len(term) ++ if delta < 0 { ++ delta = -delta ++ } ++ if d < bestDist || (d == bestDist && delta < bestDelta) { ++ bestDist = d ++ bestLen = wl ++ bestDelta = delta ++ } ++ } ++ return bestLen ++} ++ ++// fuzzyContains reports whether any substring of content is within maxDist ++// edits (true Levenshtein: insertions, deletions, and substitutions) of term. + func fuzzyContains(content, term string, maxDist, termLen int) bool { + contentLen := len(content) +- if contentLen == 0 || termLen == 0 || termLen > contentLen { ++ if contentLen == 0 || termLen == 0 { ++ return false ++ } ++ ++ minLen, maxLen := fuzzyWindowBounds(termLen, maxDist) ++ if minLen > contentLen { + return false + } + +- for i := 0; i <= contentLen-termLen; i++ { +- window := content[i : i+termLen] +- if levenshtein(window, term) <= maxDist { ++ for i := 0; i <= contentLen-minLen; i++ { ++ if bestFuzzyMatchAt(content, term, i, maxDist, minLen, maxLen) >= 0 { + return true + } + } +@@ -651,18 +701,30 @@ func fuzzyContains(content, term string, maxDist, termLen int) bool { + + // fuzzyFind finds all match locations in content that are within the given + // edit distance of the term. Returns [][]int where each entry is [start, end]. ++// Like fuzzyContains it considers variable-length windows so insertions and ++// deletions match. To avoid emitting a swarm of overlapping windows for one ++// logical match, it records the best window at each start and then advances ++// past it. + func fuzzyFind(content, term string, maxDist, termLen int) [][]int { + contentLen := len(content) +- if contentLen == 0 || termLen == 0 || termLen > contentLen { ++ if contentLen == 0 || termLen == 0 { ++ return nil ++ } ++ ++ minLen, maxLen := fuzzyWindowBounds(termLen, maxDist) ++ if minLen > contentLen { + return nil + } + + var locs [][]int +- for i := 0; i <= contentLen-termLen; i++ { +- window := content[i : i+termLen] +- if levenshtein(window, term) <= maxDist { +- locs = append(locs, []int{i, i + termLen}) +- } ++ for i := 0; i <= contentLen-minLen; { ++ wl := bestFuzzyMatchAt(content, term, i, maxDist, minLen, maxLen) ++ if wl < 0 { ++ i++ ++ continue ++ } ++ locs = append(locs, []int{i, i + wl}) ++ i += wl + } + return locs + } +diff --git a/pkg/search/search_test.go b/pkg/search/search_test.go +index 6cbd01f..eacbd75 100644 +--- a/pkg/search/search_test.go ++++ b/pkg/search/search_test.go +@@ -57,7 +57,10 @@ func TestExecutor(t *testing.T) { + {"Fuzzy Distance 1", "houss~1", false, []string{"src/main/file1.go"}}, // "houss" is distance 1 from "house" (only in file1) + {"Fuzzy Distance 1 No Match", "zzz~1", false, []string{}}, + {"Fuzzy AND Keyword", "houss~1 AND brown", false, []string{"src/main/file1.go"}}, +- {"Fuzzy Distance 2", "hovze~2", false, []string{"src/main/file1.go"}}, // "hovze" is distance 2 from "house" (u→v, s→z), only in file1 ++ {"Fuzzy Distance 2", "houze~2", false, []string{"src/main/file1.go"}}, // "houze" is distance 1 from "house" (s→z), only in file1 ++ // Mid-word insertion/deletion must match (true Levenshtein, not just same-length substitution). ++ {"Fuzzy Distance 1 deletion", "hose~1", false, []string{"src/main/file1.go"}}, // "hose" -> "house" (insert 'u'), distance 1 ++ {"Fuzzy Distance 1 insertion", "houxse~1", false, []string{"src/main/file1.go"}}, // "houxse" -> "house" (delete 'x'), distance 1 + + // Colon filter syntax + {"Colon file filter", "cat file:file1", false, []string{"src/main/file1.go"}}, diff --git a/docker/cs/luau-declarations.patch b/docker/cs/luau-declarations.patch new file mode 100644 index 0000000..794ecac --- /dev/null +++ b/docker/cs/luau-declarations.patch @@ -0,0 +1,32 @@ +Add a Luau declaration-pattern table to the cs structural ranker. + +Upstream cs (v3.1.0) ships a "Lua" entry in languageDeclarationPatterns but no +"Luau" entry, even though its bundled scc database recognises ".luau" files as a +distinct "Luau" language. Without a matching declaration table, every match in a +.luau file is classified as a plain "usage": --only-declarations returns nothing +and the structural ranker gives definitions no boost. This is the dominant file +type in Roblox codebases, so we add a Luau entry that mirrors Lua's function +prefixes and additionally covers Luau's `type` / `export type` declarations. + +This is a purely additive change (one new map entry); it does not alter any +existing language's behaviour and passes cs's own pkg/ranker test suite. Applied +during the Docker build via `git apply` against the pinned v3.1.0 checkout +(see Dockerfile / Dockerfile.dev). Candidate for upstreaming to boyter/cs. + +diff --git a/pkg/ranker/declarations.go b/pkg/ranker/declarations.go +index 42cd934..36f9f68 100644 +--- a/pkg/ranker/declarations.go ++++ b/pkg/ranker/declarations.go +@@ -187,6 +187,12 @@ var languageDeclarationPatterns = map[string][]DeclarationPattern{ + {Prefix: []byte("function ")}, + {Prefix: []byte("local function ")}, + }, ++ "Luau": { ++ {Prefix: []byte("function ")}, ++ {Prefix: []byte("local function ")}, ++ {Prefix: []byte("type ")}, ++ {Prefix: []byte("export type ")}, ++ }, + "Scala": { + {Prefix: []byte("def ")}, + {Prefix: []byte("val ")}, diff --git a/packages/api/src/agent-manager.ts b/packages/api/src/agent-manager.ts index 7af2b67..2532efa 100644 --- a/packages/api/src/agent-manager.ts +++ b/packages/api/src/agent-manager.ts @@ -20,6 +20,7 @@ import { createReadTabTool, createRetrieveTool, createRunShellTool, + createSearchCodeTool, createSendToTabTool, createSkillsWatcher, createSummonTool, @@ -80,6 +81,8 @@ const TOOL_DESCRIPTIONS: Record<string, string> = { write_file: "Write content to a file (creates parent directories if needed)", run_shell: "Execute shell commands in the working directory (bash). Returns stdout, stderr, and exit code. Set background=true to run in the background and get a job_id for later retrieval. Do NOT run destructive or irreversible commands unless the user explicitly requests them.", + search_code: + "Search the codebase by query using the 'cs' code search engine (relevance-ranked, structure-aware). Returns the most relevant files first with matching snippets and line numbers. Better than grep/find for exploratory 'where is X / how does Y work' searches; use run_shell with rg for exhaustive exact-match lists.", todo: "Create/maintain a todo list to plan and track work. Declarative whole-list write: send the entire list in `todos` each call (it replaces the previous list). Statuses: pending, in_progress, completed, cancelled.", summon: "Spawn a child agent to work on a task independently. By default blocks until the child finishes. Set background=true to return immediately with an agent_id for later retrieval.", @@ -511,10 +514,11 @@ export class AgentManager { const permSendToTab = getSetting("perm_send_to_tab") === "allow"; const permReadTab = getSetting("perm_read_tab") === "allow"; const permWebSearch = getSetting("perm_web_search") === "allow"; + const permSearchCode = getSetting("perm_search_code") === "allow"; const permYoutubeTranscribe = getSetting("perm_youtube_transcribe") === "allow"; const permLsp = getSetting("perm_lsp") === "allow"; const sysPrompt = getSetting("system_prompt") ?? ""; - const permKey = `${permRead}:${permEdit}:${permBash}:${permSummon}:${permUserAgent}:${permSendToTab}:${permReadTab}:${permWebSearch}:${permYoutubeTranscribe}:${permLsp}:${sysPrompt}`; + const permKey = `${permRead}:${permEdit}:${permBash}:${permSummon}:${permUserAgent}:${permSendToTab}:${permReadTab}:${permWebSearch}:${permYoutubeTranscribe}:${permSearchCode}:${permLsp}:${sysPrompt}`; // If the override differs or permissions changed, invalidate the cached agent if ( @@ -597,6 +601,12 @@ export class AgentManager { tool: createRunShellTool(workingDirectory, tabAgent.shellStore), }); } + if (allowed.has("search_code")) { + toolEntries.push({ + name: "search_code", + tool: createSearchCodeTool(workingDirectory), + }); + } if (allowed.has("web_search")) { toolEntries.push({ name: "web_search", tool: createWebSearchTool() }); } @@ -696,6 +706,12 @@ export class AgentManager { tool: createRunShellTool(workingDirectory, tabAgent.shellStore), }); } + if (permSearchCode) { + toolEntries.push({ + name: "search_code", + tool: createSearchCodeTool(workingDirectory), + }); + } if (permWebSearch) { toolEntries.push({ name: "web_search", tool: createWebSearchTool() }); } diff --git a/packages/api/tests/agent-manager.test.ts b/packages/api/tests/agent-manager.test.ts index 6efe15e..dbbcc65 100644 --- a/packages/api/tests/agent-manager.test.ts +++ b/packages/api/tests/agent-manager.test.ts @@ -472,6 +472,14 @@ vi.mock("@dispatch/core", () => ({ execute: async () => "mock", }; }, + createSearchCodeTool(_wd: string) { + return { + name: "search_code", + description: "search code", + parameters: { _type: "z.ZodObject", shape: {} }, + execute: async () => "mock", + }; + }, createYoutubeTranscribeTool() { return { name: "youtube_transcribe", @@ -1494,6 +1502,27 @@ describe("AgentManager", () => { }); }); + describe("search_code permission gating", () => { + // Reuses the parent-path tool construction to confirm the perm flag wires + // the search_code tool on/off correctly. + async function toolsForPerms(tabId: string, perms: Record<string, string>): Promise<string[]> { + for (const [k, v] of Object.entries(perms)) setFakeSetting(k, v); + const manager = new AgentManager(); + await manager.processMessage(tabId, "go"); + return constructedAgents.at(-1)?.toolNames ?? []; + } + + it("grants search_code when perm_search_code is allowed", async () => { + const tools = await toolsForPerms("tab-cs-on", { perm_search_code: "allow" }); + expect(tools).toContain("search_code"); + }); + + it("omits search_code when perm_search_code is not allowed", async () => { + const tools = await toolsForPerms("tab-cs-off", {}); + expect(tools).not.toContain("search_code"); + }); + }); + describe("summon / user_agent permission split", () => { // Drives the real parent-path tool construction in // getOrCreateAgentForTab by toggling perm_summon and perm_user_agent diff --git a/packages/api/tests/routes.test.ts b/packages/api/tests/routes.test.ts index 6ec8ca6..37c19ca 100644 --- a/packages/api/tests/routes.test.ts +++ b/packages/api/tests/routes.test.ts @@ -313,6 +313,14 @@ vi.mock("@dispatch/core", () => ({ execute: async () => "mock", }; }, + createSearchCodeTool(_wd: string) { + return { + name: "search_code", + description: "search code", + parameters: { _type: "z.ZodObject", shape: {} }, + execute: async () => "mock", + }; + }, createYoutubeTranscribeTool() { return { name: "youtube_transcribe", diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 8608b6a..08b426f 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -107,6 +107,7 @@ export { createReadTabTool, type ReadTabCallbacks } from "./tools/read-tab.js"; export { createToolRegistry } from "./tools/registry.js"; export { createRetrieveTool, type RetrieveCallbacks } from "./tools/retrieve.js"; export { BackgroundShellStore, createRunShellTool } from "./tools/run-shell.js"; +export { createSearchCodeTool } from "./tools/search-code.js"; export { createSendToTabTool, type ResolvedTabRef, diff --git a/packages/core/src/tools/search-code.ts b/packages/core/src/tools/search-code.ts new file mode 100644 index 0000000..4350f6a --- /dev/null +++ b/packages/core/src/tools/search-code.ts @@ -0,0 +1,362 @@ +import { spawn } from "node:child_process"; +import { stat } from "node:fs/promises"; +import { relative, sep } from "node:path"; +import { z } from "zod"; +import type { ToolDefinition } from "../types/index.js"; +import { canonicalize } from "./path-utils.js"; + +// Resolve the `cs` binary: an explicit override wins, otherwise rely on PATH. +// The deployed images build a patched, statically-linked `cs` into +// /usr/local/bin/cs (see Dockerfile); local dev can point DISPATCH_CS_BIN at a +// custom build. Read at call time so the environment can change at runtime +// (and so tests can point it at a stub or temp build). +function resolveCsBin(): string { + return process.env.DISPATCH_CS_BIN || "cs"; +} + +const DEFAULT_RESULT_LIMIT = 20; +const MAX_RESULT_LIMIT = 100; +const MAX_CONTEXT = 20; +const MIN_SNIPPET_LENGTH = 50; +const MAX_SNIPPET_LENGTH = 2000; +const TIMEOUT_MS = 30_000; +// Hard cap on any single rendered snippet line. Mirrors read-file.ts so a +// matched minified/generated line (e.g. a 2 MB bundle line) can't blow up the +// payload. The universal truncator bounds total output; this bounds per-line. +const MAX_LINE_CHARS = 500; + +/** Maps the `only` enum to the corresponding cs flag. */ +const ONLY_FLAGS: Record<string, string> = { + code: "--only-code", + comments: "--only-comments", + strings: "--only-strings", + declarations: "--only-declarations", + usages: "--only-usages", +}; + +/** One line within a cs JSON match result. */ +interface CsLine { + line_number: number; + content: string; + match_positions?: Array<[number, number]>; +} + +/** A single file result in cs `-f json` output. */ +interface CsResult { + filename: string; + location: string; + score: number; + /** Present in "lines"/"grep" snippet modes. */ + lines?: CsLine[]; + /** + * Present instead of `lines` in cs's "snippet" mode (the default "auto" + * mode selects it for prose). We force a lines-based mode (see buildFlags), + * but this is kept as a defensive fallback so a content-shape result is + * still rendered rather than shown as a bare header. + */ + content?: string; + matchlocations?: Array<[number, number]>; + language?: string; + total_lines?: number; +} + +export function createSearchCodeTool(workingDirectory: string): ToolDefinition { + return { + name: "search_code", + description: + "Search the codebase by query using `cs` (code spelunker) — a fast, relevance-ranked code search engine. " + + "Prefer this over grep/find for EXPLORATORY 'where is X / how does Y work' searches: it ranks the most " + + "relevant files first and returns matching snippets with line numbers, so you spend fewer turns and tokens. " + + "It respects .gitignore and skips hidden/binary files. " + + 'Query syntax: space-separated terms are AND\'d; supports OR, NOT, "exact phrases", fuzzy~1, /regex/, and ' + + "metadata filters like lang:Go, file:test, path:src. " + + "It is a ranked text search, NOT a semantic/LSP index: it won't resolve types or imports. For an EXHAUSTIVE " + + "list of every exact match (e.g. before a rename), use run_shell with ripgrep (rg) instead.", + parameters: z.object({ + query: z + .string() + .describe( + 'The search query. Terms are AND\'d by default. Supports OR, NOT, "phrases", fuzzy~1, /regex/, and filters like lang:Go, file:test, path:src.', + ), + path: z + .string() + .optional() + .describe( + "Subdirectory to scope the search to, relative to the working directory. Defaults to the whole working directory.", + ), + case_sensitive: z + .boolean() + .optional() + .describe("Make the search case-sensitive. Default: false (case-insensitive)."), + include_ext: z + .string() + .optional() + .describe( + 'Comma-separated list of file extensions to limit the search to (case-sensitive), e.g. "go,ts,lua".', + ), + exclude_pattern: z + .string() + .optional() + .describe( + 'Comma-separated list of path patterns to exclude (case-sensitive), e.g. "vendor,_test.go".', + ), + context: z + .number() + .int() + .min(0) + .optional() + .describe( + `Lines of context to show before and after each matching line (0-${MAX_CONTEXT}). When set, switches to a grep-style per-line window.`, + ), + result_limit: z + .number() + .int() + .min(1) + .optional() + .describe( + `Maximum number of file results to return. Default: ${DEFAULT_RESULT_LIMIT}, max: ${MAX_RESULT_LIMIT}.`, + ), + snippet_length: z + .number() + .int() + .min(MIN_SNIPPET_LENGTH) + .optional() + .describe( + `Snippet size in bytes for prose/text files (${MIN_SNIPPET_LENGTH}-${MAX_SNIPPET_LENGTH}). Has little effect on code files, which use a fixed line window — use 'context' to widen code snippets.`, + ), + only: z + .enum(["code", "comments", "strings", "declarations", "usages"]) + .optional() + .describe( + "Restrict matches structurally: code, comments, strings, declarations (definitions like func/class/type), " + + "or usages (call sites). Best-effort and language-dependent — strong for Go/TypeScript/Python/Lua/Luau, " + + "unavailable for unsupported languages (which fall back to plain text ranking).", + ), + }), + execute: async (args: Record<string, unknown>): Promise<string> => { + const query = typeof args.query === "string" ? args.query : ""; + if (query.trim() === "") { + return "Error: query is required (a non-empty string)."; + } + + // Resolve and contain the optional search path within the workdir. + // Canonicalize so a symlink-in-workdir pointing outside is detected, + // matching the containment semantics of list_files / read_file. + const relPath = asString(args.path) ?? "."; + const absoluteWorkDir = await canonicalize(workingDirectory); + const searchDir = await canonicalize(workingDirectory, relPath); + if (searchDir !== absoluteWorkDir && !searchDir.startsWith(`${absoluteWorkDir}/`)) { + return `Error: Path "${relPath}" is outside the working directory.`; + } + + // cs's --dir expects a directory; pointing it at a file silently + // returns no matches. Catch that and give an actionable hint instead + // of a misleading "No matches found". + if (relPath !== ".") { + try { + const st = await stat(searchDir); + if (!st.isDirectory()) { + return `Error: Path "${relPath}" is a file, not a directory. The 'path' parameter scopes the search to a directory; use read_file to read a single file.`; + } + } catch { + return `Error: Path "${relPath}" does not exist in the working directory.`; + } + } + + const flags = buildFlags(args, searchDir); + // `--` terminates cs flag parsing so a query that begins with "-" + // (e.g. "-hello" or "--foo") is treated as the positional search term + // rather than parsed as a (possibly invalid) cs flag. + const spawnArgs = [...flags, "--", query]; + + let stdout = ""; + let stderr = ""; + const result = await new Promise<{ + code: number | null; + signal: NodeJS.Signals | null; + error?: string; + errorCode?: string; + }>((resolve) => { + const child = spawn(resolveCsBin(), spawnArgs, { + cwd: workingDirectory, + env: process.env, + timeout: TIMEOUT_MS, + stdio: ["ignore", "pipe", "pipe"], + }); + child.stdout?.on("data", (d: Buffer) => { + stdout += d.toString(); + }); + child.stderr?.on("data", (d: Buffer) => { + stderr += d.toString(); + }); + child.on("close", (code, signal) => resolve({ code, signal })); + child.on("error", (err) => + resolve({ + code: null, + signal: null, + error: err.message, + errorCode: (err as NodeJS.ErrnoException).code, + }), + ); + }); + + if (result.error) { + // The binary is missing or not executable — give an actionable hint. + if (result.errorCode === "ENOENT" || result.error.includes("ENOENT")) { + return missingBinaryError(); + } + return `Error: failed to run cs: ${result.error}`; + } + + // A signal kill (e.g. SIGTERM from the spawn timeout) or a non-zero + // exit means cs failed — surface it (with stderr) instead of silently + // reporting "No matches found". cs exits 0 even when there are no + // matches, so a clean exit always falls through to the parsing below. + if (result.signal) { + const detail = stderr.trim() ? `\n${stderr.trim()}` : ""; + if (result.signal === "SIGTERM") { + return `Error: cs search timed out after ${TIMEOUT_MS / 1000}s. Try a narrower query or a smaller path.${detail}`; + } + return `Error: cs was terminated by signal ${result.signal}.${detail}`; + } + if (result.code !== 0) { + const detail = stderr.trim() ? `\n${stderr.trim()}` : ""; + return `Error: cs exited with code ${result.code}.${detail}`; + } + + // cs prints `null` (and exit 0) when there are no matches. + const trimmed = stdout.trim(); + if (trimmed === "" || trimmed === "null") { + return "No matches found."; + } + + let parsed: CsResult[]; + try { + parsed = JSON.parse(trimmed) as CsResult[]; + } catch { + // Couldn't parse — surface what cs produced so the caller isn't blind. + const detail = stderr.trim() ? `\nstderr: ${stderr.trim()}` : ""; + return `Error: could not parse cs output as JSON.${detail}\n\nRaw output:\n${trimmed.slice(0, 2000)}`; + } + + if (!Array.isArray(parsed) || parsed.length === 0) { + return "No matches found."; + } + + return formatResults(parsed, absoluteWorkDir); + }, + }; +} + +/** Build the cs CLI flags (everything except the trailing query). */ +function buildFlags(args: Record<string, unknown>, searchDir: string): string[] { + const flags: string[] = ["-f", "json", "--dir", searchDir]; + + if (args.case_sensitive === true) flags.push("-c"); + + const includeExt = asString(args.include_ext); + if (includeExt) flags.push("-i", includeExt); + + const excludePattern = asString(args.exclude_pattern); + if (excludePattern) flags.push("-x", excludePattern); + + // Snippet mode selection. cs's default ("auto") emits a `lines[]` array for + // code but a single `content` string for prose (.md/.html/…), which our + // renderer can't show — so prose results would come back as bare headers. + // It also ignores -C/--context entirely in auto/lines mode. + // + // - No `context` given → force "lines": every file type (code AND prose) + // returns a `lines[]` window, so prose snippets render too. + // - `context` given → use "grep": the only mode where -C actually widens + // the window; it likewise returns `lines[]` for all file types. + if (typeof args.context === "number") { + const context = clamp(Math.floor(args.context), 0, MAX_CONTEXT); + flags.push("--snippet-mode", "grep", "-C", String(context)); + } else { + flags.push("--snippet-mode", "lines"); + } + + const requestedLimit = + typeof args.result_limit === "number" + ? clamp(Math.floor(args.result_limit), 1, MAX_RESULT_LIMIT) + : DEFAULT_RESULT_LIMIT; + flags.push("--result-limit", String(requestedLimit)); + + if (typeof args.snippet_length === "number") { + const snippet = clamp(Math.floor(args.snippet_length), MIN_SNIPPET_LENGTH, MAX_SNIPPET_LENGTH); + flags.push("-n", String(snippet)); + } + + const only = asString(args.only); + if (only && ONLY_FLAGS[only]) flags.push(ONLY_FLAGS[only]); + + return flags; +} + +/** Render cs JSON results into compact, readable per-file blocks. */ +function formatResults(results: CsResult[], absoluteWorkDir: string): string { + const blocks: string[] = []; + // Match the workdir only at a path boundary so a sibling dir that merely + // shares the prefix (e.g. workdir "/app" vs "/app-secrets") isn't treated + // as nested and rendered as a "../app-secrets/..." relative path. + const workdirPrefix = absoluteWorkDir.endsWith(sep) ? absoluteWorkDir : absoluteWorkDir + sep; + for (const r of results) { + // Present paths relative to the workdir so output is portable and compact. + const insideWorkdir = r.location === absoluteWorkDir || r.location.startsWith(workdirPrefix); + const rel = insideWorkdir ? relative(absoluteWorkDir, r.location) || r.filename : r.location; + const lang = r.language ? ` [${r.language}]` : ""; + const score = typeof r.score === "number" ? ` (score ${r.score.toFixed(2)})` : ""; + const header = `${rel}${lang}${score}`; + + let body: string[]; + if (r.lines && r.lines.length > 0) { + body = r.lines.map((l) => { + const marker = l.match_positions && l.match_positions.length > 0 ? ">" : " "; + return ` ${marker} ${l.line_number}: ${truncateLine(l.content)}`; + }); + } else if (r.content && r.content.trim() !== "") { + // Fallback for cs's "snippet"-mode shape (no per-line numbers): show + // the snippet text itself so the result isn't a bare header. + body = r.content.split("\n").map((line) => ` ${truncateLine(line)}`); + } else { + body = [" (match in file; no snippet available)"]; + } + + blocks.push([header, ...body].join("\n")); + } + + const count = results.length; + const heading = `Found matches in ${count} file${count === 1 ? "" : "s"} (ranked by relevance):`; + return [heading, "", blocks.join("\n\n")].join("\n"); +} + +function clamp(n: number, min: number, max: number): number { + return Math.min(max, Math.max(min, n)); +} + +/** Cap an individual snippet line so a minified/generated line can't bloat output. */ +function truncateLine(line: string): string { + if (line.length <= MAX_LINE_CHARS) return line; + return `${line.slice(0, MAX_LINE_CHARS)}… [line truncated, ${line.length.toLocaleString()} chars]`; +} + +/** + * Coerce a tool argument to a trimmed string, or undefined. Guards against a + * model hallucinating a non-string (e.g. an array `["ts","go"]`) for a + * string-typed param: returning undefined makes the flag a no-op instead of + * throwing `x.trim is not a function` and crashing the tool call. + */ +function asString(v: unknown): string | undefined { + if (typeof v !== "string") return undefined; + const t = v.trim(); + return t === "" ? undefined : t; +} + +function missingBinaryError(): string { + return [ + "Error: search_code requires the 'cs' (code spelunker) binary, which was not found.", + "Install it with: go install github.com/boyter/cs/[email protected]", + "or set the DISPATCH_CS_BIN environment variable to the path of a cs binary.", + "(In the official Docker images cs is bundled at /usr/local/bin/cs.)", + ].join("\n"); +} diff --git a/packages/core/src/tools/summon.ts b/packages/core/src/tools/summon.ts index cfee8b8..b941152 100644 --- a/packages/core/src/tools/summon.ts +++ b/packages/core/src/tools/summon.ts @@ -186,6 +186,7 @@ export function createSummonTool( " - list_files: List files and directories", " - write_file: Write/edit files", " - run_shell: Execute shell commands", + " - search_code: Search the codebase with the cs ranked code-search engine", " - todo: Track work items", " - summon: Spawn its own child agents (enables nesting)", " - retrieve: Collect results from its children (required if summon is given)", @@ -285,6 +286,7 @@ export function createSummonTool( "list_files", "write_file", "run_shell", + "search_code", "todo", "summon", "retrieve", diff --git a/packages/core/tests/agents/loader.test.ts b/packages/core/tests/agents/loader.test.ts index 92f9877..a223a4f 100644 --- a/packages/core/tests/agents/loader.test.ts +++ b/packages/core/tests/agents/loader.test.ts @@ -43,6 +43,7 @@ describe("expandAgentToolNames", () => { "retrieve", "web_search", "youtube_transcribe", + "search_code", "send_to_tab", "read_tab", ]); @@ -52,6 +53,7 @@ describe("expandAgentToolNames", () => { "retrieve", "web_search", "youtube_transcribe", + "search_code", "send_to_tab", "read_tab", ]), diff --git a/packages/core/tests/tools/search-code.test.ts b/packages/core/tests/tools/search-code.test.ts new file mode 100644 index 0000000..c4e933c --- /dev/null +++ b/packages/core/tests/tools/search-code.test.ts @@ -0,0 +1,511 @@ +import { spawnSync } from "node:child_process"; +import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { mkdtemp as mkdtempP, rm as rmP, writeFile as writeFileP } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { createSearchCodeTool } from "../../src/tools/search-code.js"; + +// A tiny stub that impersonates `cs`: it ignores its args and prints whatever +// JSON we put in the CS_STUB_OUTPUT env var. This makes JSON→text formatting +// tests fully deterministic without needing a real cs binary in CI. +function writeStub(dir: string, body: string): string { + const stubPath = join(dir, "cs-stub.sh"); + writeFileSync(stubPath, body, { mode: 0o755 }); + chmodSync(stubPath, 0o755); + return stubPath; +} + +const ECHO_ENV_STUB = `#!/usr/bin/env bash +printf '%s' "$CS_STUB_OUTPUT" +`; + +// A stub that writes to stderr and exits non-zero, impersonating a cs failure +// (bad flag, invalid regex, etc.). +const FAIL_STUB = `#!/usr/bin/env bash +echo "cs: simulated failure on stderr" >&2 +exit 3 +`; + +describe("search_code tool", () => { + let workDir: string; + const savedBin = process.env.DISPATCH_CS_BIN; + const savedStubOut = process.env.CS_STUB_OUTPUT; + + beforeEach(async () => { + workDir = await mkdtempP(join(tmpdir(), "dispatch-cs-test-")); + }); + + afterEach(async () => { + await rmP(workDir, { recursive: true, force: true }); + if (savedBin === undefined) delete process.env.DISPATCH_CS_BIN; + else process.env.DISPATCH_CS_BIN = savedBin; + if (savedStubOut === undefined) delete process.env.CS_STUB_OUTPUT; + else process.env.CS_STUB_OUTPUT = savedStubOut; + }); + + it("exposes the expected name and schema", () => { + const tool = createSearchCodeTool(workDir); + expect(tool.name).toBe("search_code"); + expect(tool.description).toContain("cs"); + // query is required; a representative set of optional knobs exist. + const shape = (tool.parameters as unknown as { shape: Record<string, unknown> }).shape; + expect(shape.query).toBeDefined(); + expect(shape.path).toBeDefined(); + expect(shape.only).toBeDefined(); + expect(shape.result_limit).toBeDefined(); + }); + + it("requires a non-empty query", async () => { + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: " " }); + expect(out).toMatch(/^Error:/); + expect(out).toContain("query is required"); + }); + + it("does not crash when params are the wrong type (model hallucination)", async () => { + const tool = createSearchCodeTool(workDir); + // A non-string query must be rejected gracefully, not throw. + const q = await tool.execute({ query: ["a", "b"] as unknown as string }); + expect(q).toMatch(/^Error:/); + expect(q).toContain("query is required"); + // A non-string include_ext (array) must not throw "x.trim is not a function". + const stubDir = await mkdtempP(join(tmpdir(), "dispatch-cs-stub-")); + try { + process.env.DISPATCH_CS_BIN = writeStub(stubDir, ECHO_ENV_STUB); + process.env.CS_STUB_OUTPUT = "null"; + const out = await tool.execute({ + query: "x", + include_ext: ["ts", "go"] as unknown as string, + exclude_pattern: { a: 1 } as unknown as string, + }); + expect(out).toBe("No matches found."); + } finally { + await rmP(stubDir, { recursive: true, force: true }); + } + }); + + it("rejects a path outside the working directory", async () => { + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "anything", path: "../../etc" }); + expect(out).toMatch(/^Error:/); + expect(out).toContain("outside the working directory"); + }); + + it("rejects a path that points at a file, not a directory", async () => { + await writeFileP(join(workDir, "a-file.ts"), "const x = 1;\n"); + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "x", path: "a-file.ts" }); + expect(out).toMatch(/^Error:/); + expect(out).toContain("is a file, not a directory"); + }); + + it("rejects a path that does not exist", async () => { + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "x", path: "no/such/dir" }); + expect(out).toMatch(/^Error:/); + expect(out).toContain("does not exist"); + }); + + it("returns an actionable error when the cs binary is missing", async () => { + process.env.DISPATCH_CS_BIN = "/nonexistent/path/to/cs-binary-xyz"; + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "anything" }); + expect(out).toMatch(/^Error:/); + expect(out).toContain("requires the 'cs'"); + expect(out).toContain("DISPATCH_CS_BIN"); + }); + + it("reports no matches when cs outputs null", async () => { + const stubDir = await mkdtempP(join(tmpdir(), "dispatch-cs-stub-")); + try { + process.env.DISPATCH_CS_BIN = writeStub(stubDir, ECHO_ENV_STUB); + process.env.CS_STUB_OUTPUT = "null"; + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "nothinghere" }); + expect(out).toBe("No matches found."); + } finally { + await rmP(stubDir, { recursive: true, force: true }); + } + }); + + it("formats cs JSON results into readable per-file blocks", async () => { + const stubDir = await mkdtempP(join(tmpdir(), "dispatch-cs-stub-")); + try { + const csJson = JSON.stringify([ + { + filename: "web-search.ts", + location: join(workDir, "packages/core/src/tools/web-search.ts"), + score: 5.24, + language: "TypeScript", + total_lines: 106, + lines: [ + { line_number: 7, content: "" }, + { + line_number: 8, + content: "export function createWebSearchTool(): ToolDefinition {", + match_positions: [[16, 35]], + }, + { line_number: 9, content: "\treturn {" }, + ], + }, + { + filename: "index.ts", + location: join(workDir, "packages/core/src/index.ts"), + score: 1.1, + language: "TypeScript", + lines: [{ line_number: 113, content: 'export { createWebSearchTool } from "./web.js";' }], + }, + ]); + process.env.DISPATCH_CS_BIN = writeStub(stubDir, ECHO_ENV_STUB); + process.env.CS_STUB_OUTPUT = csJson; + + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "createWebSearchTool" }); + + expect(out).toContain("Found matches in 2 files"); + // Paths are rendered relative to the workdir. + expect(out).toContain("packages/core/src/tools/web-search.ts [TypeScript] (score 5.24)"); + expect(out).not.toContain(workDir); + // Matched line is marked with '>'; line numbers + content present. + expect(out).toContain("> 8: export function createWebSearchTool(): ToolDefinition {"); + expect(out).toContain(" 7: "); + expect(out).toContain("packages/core/src/index.ts [TypeScript] (score 1.10)"); + } finally { + await rmP(stubDir, { recursive: true, force: true }); + } + }); + + it("renders cs 'content'-shape (prose) results instead of a bare header", async () => { + const stubDir = await mkdtempP(join(tmpdir(), "dispatch-cs-stub-")); + try { + // cs's snippet mode emits `content` + `matchlocations` and no `lines`. + const csJson = JSON.stringify([ + { + filename: "notes.md", + location: join(workDir, "docs/notes.md"), + score: 0.42, + language: "Markdown", + content: "Some heading\nthe orchestration paragraph that matched", + matchlocations: [[13, 26]], + }, + ]); + process.env.DISPATCH_CS_BIN = writeStub(stubDir, ECHO_ENV_STUB); + process.env.CS_STUB_OUTPUT = csJson; + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "orchestration" }); + expect(out).toContain("docs/notes.md [Markdown] (score 0.42)"); + // The snippet text must be present, not a bare header. + expect(out).toContain("the orchestration paragraph that matched"); + expect(out).not.toContain("no snippet available"); + } finally { + await rmP(stubDir, { recursive: true, force: true }); + } + }); + + it("truncates an excessively long snippet line", async () => { + const stubDir = await mkdtempP(join(tmpdir(), "dispatch-cs-stub-")); + try { + const longContent = `const x = "${"Z".repeat(5000)}";`; + const csJson = JSON.stringify([ + { + filename: "big.ts", + location: join(workDir, "big.ts"), + score: 1, + language: "TypeScript", + lines: [{ line_number: 1, content: longContent, match_positions: [[10, 14]] }], + }, + ]); + process.env.DISPATCH_CS_BIN = writeStub(stubDir, ECHO_ENV_STUB); + process.env.CS_STUB_OUTPUT = csJson; + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "x" }); + expect(out).toContain("line truncated"); + // No single output line should approach the raw 5k length. + const longest = Math.max(...out.split("\n").map((l) => l.length)); + expect(longest).toBeLessThan(700); + } finally { + await rmP(stubDir, { recursive: true, force: true }); + } + }); + + it("surfaces raw output when cs returns unparseable JSON", async () => { + const stubDir = await mkdtempP(join(tmpdir(), "dispatch-cs-stub-")); + try { + process.env.DISPATCH_CS_BIN = writeStub(stubDir, ECHO_ENV_STUB); + process.env.CS_STUB_OUTPUT = "this is not json"; + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "x" }); + expect(out).toMatch(/^Error:/); + expect(out).toContain("could not parse cs output"); + expect(out).toContain("this is not json"); + } finally { + await rmP(stubDir, { recursive: true, force: true }); + } + }); + + it("reports an error (not 'No matches') when cs exits non-zero", async () => { + const stubDir = await mkdtempP(join(tmpdir(), "dispatch-cs-stub-")); + try { + process.env.DISPATCH_CS_BIN = writeStub(stubDir, FAIL_STUB); + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "x" }); + expect(out).toMatch(/^Error:/); + expect(out).toContain("exited with code 3"); + // stderr from cs is surfaced to the caller. + expect(out).toContain("simulated failure on stderr"); + expect(out).not.toContain("No matches found"); + } finally { + await rmP(stubDir, { recursive: true, force: true }); + } + }); + + // ── Live integration: only runs when a real `cs` binary is available. ── + const liveCsBin = findRealCs(); + describe.runIf(liveCsBin)("live cs binary", () => { + it("finds a real match and ranks the defining file", async () => { + process.env.DISPATCH_CS_BIN = liveCsBin as string; + // Seed a small tree with a clear match. + await writeFileP( + join(workDir, "alpha.ts"), + "export function findTheNeedle() {\n return 42;\n}\n", + ); + await writeFileP(join(workDir, "beta.ts"), "const x = 1;\n// nothing relevant here\n"); + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "findTheNeedle" }); + expect(out).toContain("alpha.ts"); + expect(out).toContain("findTheNeedle"); + expect(out).not.toContain("Error:"); + }); + + it("treats a dash-leading query as a search term, not a cs flag", async () => { + process.env.DISPATCH_CS_BIN = liveCsBin as string; + // A literal token beginning with '-' must not be parsed as a flag. + await writeFileP(join(workDir, "dash.ts"), "const dashToken = 1;\n"); + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "-dashToken" }); + // Whether or not cs ranks a hit, it must NOT error out on flag parsing. + expect(out).not.toContain("unknown shorthand flag"); + expect(out).not.toMatch(/^Error: cs exited/); + }); + + it("renders snippet lines for prose (markdown) matches", async () => { + process.env.DISPATCH_CS_BIN = liveCsBin as string; + await writeFileP( + join(workDir, "doc.md"), + "# Title\n\nThis paragraph mentions widgetronics in prose.\n", + ); + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "widgetronics" }); + expect(out).toContain("doc.md"); + // The matching prose text must be shown, not just a bare header. + expect(out).toContain("widgetronics"); + expect(out).not.toContain("no snippet available"); + }); + + it("widens the snippet window when context is given", async () => { + process.env.DISPATCH_CS_BIN = liveCsBin as string; + const body = Array.from({ length: 21 }, (_, i) => `line ${i + 1}`); + body[10] = "const findContextTarget = 1;"; + await writeFileP(join(workDir, "ctx.ts"), `${body.join("\n")}\n`); + const tool = createSearchCodeTool(workDir); + const countSnippetLines = (s: string) => + s.split("\n").filter((l) => /^\s+>?\s*\d+:/.test(l)).length; + const narrow = await tool.execute({ + query: "findContextTarget", + context: 0, + result_limit: 1, + }); + const wide = await tool.execute({ + query: "findContextTarget", + context: 6, + result_limit: 1, + }); + expect(countSnippetLines(wide)).toBeGreaterThan(countSnippetLines(narrow)); + }); + + it("returns 'No matches found.' for a query with no hits", async () => { + process.env.DISPATCH_CS_BIN = liveCsBin as string; + await writeFileP(join(workDir, "alpha.ts"), "export const a = 1;\n"); + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "zzz_nonexistent_token_qqq" }); + expect(out).toBe("No matches found."); + }); + + it("tags .luau files as Luau", async () => { + process.env.DISPATCH_CS_BIN = liveCsBin as string; + await writeFileP(join(workDir, "mod.luau"), "function Mod.doThing()\n\treturn 1\nend\n"); + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "doThing" }); + expect(out).toContain("mod.luau"); + expect(out).toContain("[Luau]"); + }); + }); + + // ── Luau declaration detection: needs a cs built with the Luau patch + // (docker/cs/luau-declarations.patch). Skipped on an unpatched/older cs. ── + const luauCsBin = findLuauCapableCs(liveCsBin); + describe.runIf(luauCsBin)("live cs binary (Luau declaration patch)", () => { + // A small Luau module exercising every declaration form the patch adds. + const LUAU_MODULE = [ + "local Mod = {}", + "", + "export type StuntResult = {", + "\tscore: number,", + "}", + "", + "type LaunchConfig = StuntResult", + "", + "function Mod.getDefaults(): LaunchConfig", + "\treturn { score = 0 }", + "end", + "", + "local function helperThing(x: number): number", + "\treturn x + 1", + "end", + "", + "Mod.live = Mod.getDefaults()", + "local used = helperThing(1)", + "", + ].join("\n"); + + beforeEach(async () => { + process.env.DISPATCH_CS_BIN = luauCsBin as string; + await writeFileP(join(workDir, "Mod.luau"), LUAU_MODULE); + }); + + it("detects `function Mod.x` declarations in .luau files", async () => { + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "getDefaults", only: "declarations" }); + expect(out).toContain("Mod.luau"); + expect(out).toContain("function Mod.getDefaults"); + expect(out).not.toContain("No matches found"); + }); + + it("detects `local function` declarations in .luau files", async () => { + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "helperThing", only: "declarations" }); + expect(out).toContain("Mod.luau"); + expect(out).toContain("local function helperThing"); + }); + + it("detects `type` / `export type` declarations in .luau files", async () => { + const tool = createSearchCodeTool(workDir); + const exportType = await tool.execute({ query: "StuntResult", only: "declarations" }); + expect(exportType).toContain("export type StuntResult"); + const aliasType = await tool.execute({ query: "LaunchConfig", only: "declarations" }); + expect(aliasType).toContain("type LaunchConfig"); + }); + + it("excludes declaration lines when only=usages", async () => { + const tool = createSearchCodeTool(workDir); + const out = await tool.execute({ query: "getDefaults", only: "usages" }); + // The call site is a usage; the `function Mod.getDefaults` definition is not. + expect(out).toContain("Mod.live = Mod.getDefaults()"); + expect(out).not.toContain("function Mod.getDefaults"); + }); + }); + + // ── Fuzzy mid-word matching: needs a cs built with the fuzzy patch + // (docker/cs/fuzzy-distance.patch). Skipped on an unpatched/older cs. ── + const fuzzyCsBin = findFuzzyCapableCs(liveCsBin); + describe.runIf(fuzzyCsBin)("live cs binary (fuzzy edit-distance patch)", () => { + beforeEach(() => { + process.env.DISPATCH_CS_BIN = fuzzyCsBin as string; + }); + + it("matches a mid-word deletion within distance 1", async () => { + await writeFileP( + join(workDir, "phys.ts"), + "export function computeSlipAngle() {\n\treturn 0;\n}\n", + ); + const tool = createSearchCodeTool(workDir); + // "computSlipAngle" drops the 'e' mid-word — edit distance 1. + const out = await tool.execute({ query: "computSlipAngle~1" }); + expect(out).toContain("phys.ts"); + expect(out).toContain("computeSlipAngle"); + expect(out).not.toBe("No matches found."); + }); + + it("matches a mid-word insertion within distance 1", async () => { + await writeFileP(join(workDir, "tire.ts"), "const tireFriction = 1;\n"); + const tool = createSearchCodeTool(workDir); + // "tireFricction" has an extra 'c' — edit distance 1. + const out = await tool.execute({ query: "tireFricction~1" }); + expect(out).toContain("tire.ts"); + expect(out).toContain("tireFriction"); + }); + }); +}); + +/** + * Locate a usable `cs` binary for live tests. Honors DISPATCH_CS_TEST_BIN, then + * a `cs` on PATH. Returns null when none is runnable, so the live suite is + * skipped rather than failing in environments without cs. + */ +function findRealCs(): string | null { + const candidates = [process.env.DISPATCH_CS_TEST_BIN, "cs"].filter(Boolean) as string[]; + for (const bin of candidates) { + try { + const res = spawnSync(bin, ["--version"], { stdio: "ignore" }); + if (res.status === 0) return bin; + } catch { + // try next + } + } + return null; +} + +/** + * Probe a `cs` binary against a throwaway corpus and return its trimmed stdout + * (or "" on any failure). Used by the capability gates below so patch-dependent + * live tests run only on a cs that actually has the patch — and skip (not fail) + * on an unpatched/older binary. + */ +function probeCs(bin: string, files: Record<string, string>, args: string[]): string { + let dir: string | undefined; + try { + dir = mkdtempSync(join(tmpdir(), "dispatch-cs-probe-")); + for (const [name, body] of Object.entries(files)) { + writeFileSync(join(dir, name), body); + } + const res = spawnSync(bin, ["-f", "json", "--dir", dir, ...args], { + encoding: "utf8", + }); + if (res.status !== 0 || !res.stdout) return ""; + return res.stdout.trim(); + } catch { + return ""; + } finally { + if (dir) rmSync(dir, { recursive: true, force: true }); + } +} + +/** + * Return the cs binary only if it recognises Luau declarations (i.e. was built + * with docker/cs/luau-declarations.patch): a `--only-declarations` search for a + * top-level `function` in a .luau file yields a result. Otherwise null → skip. + */ +function findLuauCapableCs(bin: string | null): string | null { + if (!bin) return null; + const out = probeCs(bin, { "probe.luau": "function Probe.thing()\n\treturn 1\nend\n" }, [ + "--only-declarations", + "--", + "thing", + ]); + return out !== "" && out !== "null" ? bin : null; +} + +/** + * Return the cs binary only if its fuzzy matcher honours mid-word edits (i.e. + * was built with docker/cs/fuzzy-distance.patch): a distance-1 deletion matches. + * Otherwise null → skip. + */ +function findFuzzyCapableCs(bin: string | null): string | null { + if (!bin) return null; + const out = probeCs(bin, { "probe.txt": "const x = computeSlipAngle;\n" }, [ + "--", + "computSlipAngle~1", + ]); + return out !== "" && out !== "null" ? bin : null; +} diff --git a/packages/frontend/src/lib/components/ToolPermissions.svelte b/packages/frontend/src/lib/components/ToolPermissions.svelte index 77452bf..6b09a07 100644 --- a/packages/frontend/src/lib/components/ToolPermissions.svelte +++ b/packages/frontend/src/lib/components/ToolPermissions.svelte @@ -48,6 +48,11 @@ const toolPermissions: ToolPermission[] = [ description: "Allow the AI to fetch YouTube video transcripts", }, { + id: "search_code", + label: "Search code", + description: "Allow the AI to search the codebase with the cs ranked code-search engine", + }, + { id: "lsp", label: "LSP queries", description: diff --git a/packages/frontend/src/lib/settings.svelte.ts b/packages/frontend/src/lib/settings.svelte.ts index ff12d69..0da4e45 100644 --- a/packages/frontend/src/lib/settings.svelte.ts +++ b/packages/frontend/src/lib/settings.svelte.ts @@ -14,6 +14,7 @@ let toolPerms = $state<Record<string, boolean>>({ external_directory: false, web_search: false, youtube_transcribe: false, + search_code: false, lsp: false, }); let savedToolPerms = $state<Record<string, boolean>>({ @@ -27,6 +28,7 @@ let savedToolPerms = $state<Record<string, boolean>>({ external_directory: false, web_search: false, youtube_transcribe: false, + search_code: false, lsp: false, }); let skillChecks = $state<Record<string, boolean>>({}); diff --git a/packaging/PKGBUILD b/packaging/PKGBUILD index a9a5b5c..878b15f 100644 --- a/packaging/PKGBUILD +++ b/packaging/PKGBUILD @@ -11,13 +11,17 @@ # bin/build-pkg # makepkg -fd in packaging/ pkgbase=dispatch -pkgname=('dispatch' 'dispatch-systemd' 'dispatch-s6') +pkgname=('dispatch' 'dispatch-systemd' 'dispatch-s6' 'code-search') pkgver=0.0.1 pkgrel=1 arch=('x86_64') url='https://github.com/anomalyco/dispatch' license=('MIT') -makedepends=('bun') +makedepends=('bun' 'go' 'git') + +# Pinned cs (code spelunker) commit (tag v3.1.0) built for the search_code tool. +# Kept in lockstep with the Docker build (see Dockerfile / Dockerfile.dev). +_cs_commit=697e0bf194bbc7a4a877e5170c70618989fc92e7 # All static files are read directly from ${_projectdir}/packaging/. We don't # use source=() because two of the s6 files share basenames (run, type), which # would collide inside ${srcdir}. @@ -56,6 +60,27 @@ build() { # Slim node_modules for runtime rm -rf node_modules bun install --frozen-lockfile --production + + # --- Build the patched `cs` code-search binary for the search_code tool --- + # Clone the pinned cs commit, apply the Luau declaration + fuzzy-distance + # patches, and build a + # statically-linked binary. cs vendors its deps, so `go build -mod=vendor` + # needs no network beyond the clone. Mirrors the Docker cs-builder stage. + # + # rm -rf first so a rerun (makepkg -e, or two invocations without -C) that + # reuses $srcdir doesn't abort on "destination path already exists". This + # PKGBUILD intentionally avoids source=() (the s6 service files share + # basenames and would collide in $srcdir), so we clone here rather than via + # makepkg's VCS source handling. + rm -rf "${srcdir}/cs-src" + git clone https://github.com/boyter/cs.git "${srcdir}/cs-src" + cd "${srcdir}/cs-src" + git checkout "${_cs_commit}" + git apply "${_projectdir}/docker/cs/luau-declarations.patch" + git apply "${_projectdir}/docker/cs/fuzzy-distance.patch" + CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPATH="${srcdir}/gopath" GOCACHE="${srcdir}/gocache" \ + go build -ldflags="-s -w" -o "${srcdir}/cs" . + "${srcdir}/cs" --version } # ---------------------------------------------------------------------------- @@ -207,3 +232,30 @@ package_dispatch-s6() { install -Dm644 "${_packagingdir}/s6/dispatch-frontend-log/notification-fd" \ "${pkgdir}/etc/s6/sv/dispatch-frontend-log/notification-fd" } + + +# ---------------------------------------------------------------------------- +# code-search — patched `cs` (code spelunker) binary +# +# A standalone, relevance-ranked code search CLI (github.com/boyter/cs), built +# from a pinned commit with an added Luau declaration table (see +# docker/cs/luau-declarations.patch). Powers Dispatch's `search_code` agent +# tool, but is a self-contained binary usable on its own. Built once in build() +# and installed to /usr/bin/cs. +# ---------------------------------------------------------------------------- +package_code-search() { + pkgdesc='code spelunker (cs) — fast, relevance-ranked code search CLI (patched for Luau)' + depends=() + # Both this package and the unrelated AUR `cs` (a colored ls) own /usr/bin/cs; + # declare the conflict so pacman reports it cleanly instead of a raw file + # collision. We do NOT `provides=('cs')` — this is a different program. + conflicts=('cs') + url='https://github.com/boyter/cs' + + install -Dm755 "${srcdir}/cs" "${pkgdir}/usr/bin/cs" + + if [ -f "${srcdir}/cs-src/LICENSE" ]; then + install -Dm644 "${srcdir}/cs-src/LICENSE" \ + "${pkgdir}/usr/share/licenses/code-search/LICENSE" + fi +} |
