diff options
| -rw-r--r-- | packages/opencode/AGENTS.md | 4 | ||||
| -rw-r--r-- | packages/opencode/specs/effect/http-api.md | 137 | ||||
| -rw-r--r-- | packages/opencode/specs/effect/migration.md (renamed from packages/opencode/specs/effect-migration.md) | 97 | ||||
| -rw-r--r-- | packages/opencode/specs/effect/routes.md | 66 | ||||
| -rw-r--r-- | packages/opencode/specs/effect/schema.md | 99 | ||||
| -rw-r--r-- | packages/opencode/specs/effect/tools.md | 96 | ||||
| -rw-r--r-- | packages/opencode/src/cli/cmd/debug/ripgrep.ts | 19 | ||||
| -rw-r--r-- | packages/opencode/src/file/ripgrep.ts | 151 | ||||
| -rw-r--r-- | packages/opencode/src/server/instance/file.ts | 11 | ||||
| -rw-r--r-- | packages/opencode/src/tool/grep.ts | 141 | ||||
| -rw-r--r-- | packages/opencode/test/file/ripgrep.test.ts | 32 | ||||
| -rw-r--r-- | packages/opencode/test/tool/grep.test.ts | 160 |
12 files changed, 665 insertions, 348 deletions
diff --git a/packages/opencode/AGENTS.md b/packages/opencode/AGENTS.md index a1291f647..fd9e597e7 100644 --- a/packages/opencode/AGENTS.md +++ b/packages/opencode/AGENTS.md @@ -13,7 +13,7 @@ Use these rules when writing or migrating Effect code. -See `specs/effect-migration.md` for the compact pattern reference and examples. +See `specs/effect/migration.md` for the compact pattern reference and examples. ## Core @@ -51,7 +51,7 @@ See `specs/effect-migration.md` for the compact pattern reference and examples. ## Effect.cached for deduplication -Use `Effect.cached` when multiple concurrent callers should share a single in-flight computation rather than storing `Fiber | undefined` or `Promise | undefined` manually. See `specs/effect-migration.md` for the full pattern. +Use `Effect.cached` when multiple concurrent callers should share a single in-flight computation rather than storing `Fiber | undefined` or `Promise | undefined` manually. See `specs/effect/migration.md` for the full pattern. ## Instance.bind — ALS for native callbacks diff --git a/packages/opencode/specs/effect/http-api.md b/packages/opencode/specs/effect/http-api.md new file mode 100644 index 000000000..a18d805a3 --- /dev/null +++ b/packages/opencode/specs/effect/http-api.md @@ -0,0 +1,137 @@ +# HttpApi migration + +Practical notes for an eventual migration of `packages/opencode` server routes from the current Hono handlers to Effect `HttpApi`, either as a full replacement or as a parallel surface. + +## Goal + +Use Effect `HttpApi` where it gives us a better typed contract for: + +- route definition +- request decoding and validation +- typed success and error responses +- OpenAPI generation +- handler composition inside Effect + +This should be treated as a later-stage HTTP boundary migration, not a prerequisite for ongoing service, route-handler, or schema work. + +## Core model + +`HttpApi` is definition-first. + +- `HttpApi` is the root API +- `HttpApiGroup` groups related endpoints +- `HttpApiEndpoint` defines a single route and its request / response schemas +- handlers are implemented separately from the contract + +This is a better fit once route inputs and outputs are already moving toward Effect Schema-first models. + +## Why it is relevant here + +The current route-effectification work is already pushing handlers toward: + +- one `AppRuntime.runPromise(Effect.gen(...))` body +- yielding services from context +- using typed Effect errors instead of Promise wrappers + +That work is a good prerequisite for `HttpApi`. Once the handler body is already a composed Effect, the remaining migration is mostly about replacing the Hono route declaration and validator layer. + +## What HttpApi gives us + +### Contracts + +Request params, query, payload, success payloads, and typed error payloads are declared in one place using Effect Schema. + +### Validation and decoding + +Incoming data is decoded through Effect Schema instead of hand-maintained Zod validators per route. + +### OpenAPI + +`HttpApi` can derive OpenAPI from the API definition, which overlaps with the current `describeRoute(...)` and `resolver(...)` pattern. + +### Typed errors + +`Schema.TaggedErrorClass` maps naturally to endpoint error contracts. + +## Likely fit for opencode + +Best fit first: + +- JSON request / response endpoints +- route groups that already mostly delegate into services +- endpoints whose request and response models can be defined with Effect Schema + +Harder / later fit: + +- SSE endpoints +- websocket endpoints +- streaming handlers +- routes with heavy Hono-specific middleware assumptions + +## Current blockers and gaps + +### Schema split + +Many route boundaries still use Zod-first validators. That does not block all experimentation, but full `HttpApi` adoption is easier after the domain and boundary types are more consistently Schema-first with `.zod` compatibility only where needed. + +### Mixed handler styles + +Many current `server/instance/*.ts` handlers still call async facades directly. Migrating those to composed `Effect.gen(...)` handlers is the low-risk step to do first. + +### Non-JSON routes + +The server currently includes SSE, websocket, and streaming-style endpoints. Those should not be the first `HttpApi` targets. + +### Existing Hono integration + +The current server composition, middleware, and docs flow are Hono-centered today. That suggests a parallel or incremental adoption plan is safer than a flag day rewrite. + +## Recommended strategy + +### 1. Finish the prerequisites first + +- continue route-handler effectification in `server/instance/*.ts` +- continue schema migration toward Effect Schema-first DTOs and errors +- keep removing service facades + +### 2. Start with one parallel group + +Introduce one small `HttpApi` group for plain JSON endpoints only. Good initial candidates are the least stateful endpoints in: + +- `server/instance/question.ts` +- `server/instance/provider.ts` +- `server/instance/permission.ts` + +Avoid `session.ts`, SSE, websocket, and TUI-facing routes first. + +### 3. Reuse existing services + +Do not re-architect business logic during the HTTP migration. `HttpApi` handlers should call the same Effect services already used by the Hono handlers. + +### 4. Run in parallel before replacing + +Prefer mounting an experimental `HttpApi` surface alongside the existing Hono routes first. That lowers migration risk and lets us compare: + +- handler ergonomics +- OpenAPI output +- auth and middleware integration +- test ergonomics + +### 5. Migrate JSON route groups gradually + +If the parallel slice works well, migrate additional JSON route groups one at a time. Leave streaming-style endpoints on Hono until there is a clear reason to move them. + +## Proposed first steps + +- [ ] add one small spike that defines an `HttpApi` group for a simple JSON route set +- [ ] use Effect Schema request / response types for that slice +- [ ] keep the underlying service calls identical to the current handlers +- [ ] compare generated OpenAPI against the current Hono/OpenAPI setup +- [ ] document how auth, instance lookup, and error mapping would compose in the new stack +- [ ] decide after the spike whether `HttpApi` should stay parallel, replace only some groups, or become the long-term default + +## Rule of thumb + +Do not start with the hardest route file. + +If `HttpApi` is adopted here, it should arrive after the handler body is already Effect-native and after the relevant request / response models have moved to Effect Schema. diff --git a/packages/opencode/specs/effect-migration.md b/packages/opencode/specs/effect/migration.md index 26b4bc624..21e222090 100644 --- a/packages/opencode/specs/effect-migration.md +++ b/packages/opencode/specs/effect/migration.md @@ -230,55 +230,9 @@ Still open at the service-shape level: - [ ] `SyncEvent` — `sync/index.ts` (deferred pending sync with James) - [ ] `Workspace` — `control-plane/workspace.ts` (deferred pending sync with James) -## Tool interface → Effect +## Tool migration -`Tool.Def.execute` and `Tool.Info.init` already return `Effect` on this branch, and the current tools in `src/tool/*.ts` have been migrated to the Effect-native `Tool.define(...)` shape. - -The remaining work here is follow-on cleanup rather than the top-level tool interface migration: - -1. Remove internal `Effect.promise(...)` bridges where practical -2. Keep replacing raw platform helpers with Effect services inside tool bodies -3. Update remaining callers and tests to prefer `yield* info.init()` / `Tool.init(...)` over older Promise-oriented patterns - -### Tool migration details - -With `Tool.Info.init()` now effectful, use this transitional pattern for migrated tools that still need Promise-based boundaries internally: - -- `Tool.defineEffect(...)` should `yield*` the services the tool depends on and close over them in the returned tool definition. -- Keep the bridge at the Promise boundary only inside the tool body when required by external APIs. Do not return Promise-based init callbacks from `Tool.define()`. -- If a tool starts requiring new services, wire them into `ToolRegistry.defaultLayer` so production callers resolve the same dependencies as tests. - -Tool tests should use the existing Effect helpers in `packages/opencode/test/lib/effect.ts`: - -- Use `testEffect(...)` / `it.live(...)` instead of creating fake local wrappers around effectful tools. -- Yield the real tool export, then initialize it: `const info = yield* ReadTool`, `const tool = yield* info.init()`. -- Run tests inside a real instance with `provideTmpdirInstance(...)` or `provideInstance(tmpdirScoped(...))` so instance-scoped services resolve exactly as they do in production. - -This keeps migrated tool tests aligned with the production service graph today, and makes the eventual `Tool.Info` → `Effect` cleanup mostly mechanical later. - -Individual tools, ordered by value: - -- [x] `apply_patch.ts` — HIGH: multi-step orchestration, error accumulation, Bus events -- [x] `bash.ts` — HIGH: shell orchestration, quoting, timeout handling, output capture -- [x] `read.ts` — HIGH: effectful interface migrated; still has raw fs/readline internals tracked below -- [x] `edit.ts` — HIGH: multi-step diff/format/publish pipeline, FileWatcher lock -- [x] `grep.ts` — MEDIUM: spawns ripgrep → ChildProcessSpawner, timeout handling -- [x] `write.ts` — MEDIUM: permission checks, diagnostics polling, Bus events -- [x] `codesearch.ts` — MEDIUM: HTTP + SSE + manual timeout → HttpClient + Effect.timeout -- [x] `webfetch.ts` — MEDIUM: fetch with UA retry, size limits → HttpClient -- [x] `websearch.ts` — MEDIUM: MCP over HTTP → HttpClient -- [x] `task.ts` — MEDIUM: task state management -- [x] `ls.ts` — MEDIUM: bounded directory listing over ripgrep-backed traversal -- [x] `multiedit.ts` — MEDIUM: sequential edit orchestration over `edit.ts` -- [x] `glob.ts` — LOW: simple async generator -- [x] `lsp.ts` — LOW: dispatch switch over LSP operations -- [x] `question.ts` — LOW: prompt wrapper -- [x] `skill.ts` — LOW: skill tool adapter -- [x] `todo.ts` — LOW: todo persistence wrapper -- [x] `invalid.ts` — LOW: invalid-tool fallback -- [x] `plan.ts` — LOW: plan file operations - -`batch.ts` was removed from `src/tool/` and is no longer tracked here. +Tool-specific migration guidance and checklist live in `tools.md`. ## Effect service adoption in already-migrated code @@ -298,11 +252,7 @@ Some already-effectified areas still use raw `Filesystem.*` or `Process.spawn` i `util/filesystem.ts` is still used widely across `src/`, and raw `fs` / `fs/promises` imports still exist in multiple tooling and infrastructure files. As services and tools are effectified, they should switch from `Filesystem.*` to yielding `AppFileSystem.Service` where possible — this should happen naturally during each migration, not as a separate sweep. -Current raw fs users that will convert during tool migration: - -- `tool/read.ts` — fs.createReadStream, readline -- `file/ripgrep.ts` — fs/promises -- `patch/index.ts` — fs, fs/promises +Tool-specific filesystem cleanup notes live in `tools.md`. ## Primitives & utilities @@ -344,47 +294,14 @@ For each service, the migration is roughly: - `ShareNext` — migrated 2026-04-11. Swapped remaining async callers to `AppRuntime.runPromise(ShareNext.Service.use(...))`, removed the `makeRuntime(...)` facade, and kept instance bootstrap on the shared app runtime. - `SessionTodo` — migrated 2026-04-10. Already matched the target service shape in `session/todo.ts`: single namespace, traced Effect methods, and no `makeRuntime(...)` facade remained; checklist updated to reflect the completed migration. - `Storage` — migrated 2026-04-10. One production caller (`Session.diff`) and all storage.test.ts tests converted to effectful style. Facades and `makeRuntime` removed. -- `SessionRunState` — migrated 2026-04-11. Single caller in `server/routes/session.ts` converted; facade removed. -- `Account` — migrated 2026-04-11. Callers in `server/routes/experimental.ts` and `cli/cmd/account.ts` converted; facade removed. +- `SessionRunState` — migrated 2026-04-11. Single caller in `server/instance/session.ts` converted; facade removed. +- `Account` — migrated 2026-04-11. Callers in `server/instance/experimental.ts` and `cli/cmd/account.ts` converted; facade removed. - `Instruction` — migrated 2026-04-11. Test-only callers converted; facade removed. - `FileTime` — migrated 2026-04-11. Test-only callers converted; facade removed. - `FileWatcher` — migrated 2026-04-11. Callers in `project/bootstrap.ts` and test converted; facade removed. -- `Question` — migrated 2026-04-11. Callers in `server/routes/question.ts` and test converted; facade removed. +- `Question` — migrated 2026-04-11. Callers in `server/instance/question.ts` and test converted; facade removed. - `Truncate` — migrated 2026-04-11. Caller in `tool/tool.ts` and test converted; facade removed. ## Route handler effectification -Route handlers should wrap their entire body in a single `AppRuntime.runPromise(Effect.gen(...))` call, yielding services from context rather than calling facades one-by-one. This eliminates multiple `runPromise` round-trips and lets handlers compose naturally. - -```ts -// Before — one facade call per service -;async (c) => { - await SessionRunState.assertNotBusy(id) - await Session.removeMessage({ sessionID: id, messageID }) - return c.json(true) -} - -// After — one Effect.gen, yield services from context -;async (c) => { - await AppRuntime.runPromise( - Effect.gen(function* () { - const state = yield* SessionRunState.Service - const session = yield* Session.Service - yield* state.assertNotBusy(id) - yield* session.removeMessage({ sessionID: id, messageID }) - }), - ) - return c.json(true) -} -``` - -When migrating, always use `{ concurrency: "unbounded" }` with `Effect.all` — route handlers should run independent service calls in parallel, not sequentially. - -Route files to convert (each handler that calls facades should be wrapped): - -- [ ] `server/routes/session.ts` — heaviest; uses Session, SessionPrompt, SessionRevert, SessionCompaction, SessionShare, SessionSummary, SessionRunState, Agent, Permission, Bus -- [ ] `server/routes/global.ts` — uses Config, Project, Provider, Vcs, Snapshot, Agent -- [ ] `server/routes/provider.ts` — uses Provider, Auth, Config -- [ ] `server/routes/question.ts` — uses Question -- [ ] `server/routes/pty.ts` — uses Pty -- [ ] `server/routes/experimental.ts` — uses Account, ToolRegistry, Agent, MCP, Config +Route-handler migration guidance and checklist live in `routes.md`. diff --git a/packages/opencode/specs/effect/routes.md b/packages/opencode/specs/effect/routes.md new file mode 100644 index 000000000..f6a61d234 --- /dev/null +++ b/packages/opencode/specs/effect/routes.md @@ -0,0 +1,66 @@ +# Route handler effectification + +Practical reference for converting server route handlers in `packages/opencode` to a single `AppRuntime.runPromise(Effect.gen(...))` body. + +## Goal + +Route handlers should wrap their entire body in a single `AppRuntime.runPromise(Effect.gen(...))` call, yielding services from context rather than calling facades one-by-one. + +This eliminates multiple `runPromise` round-trips and lets handlers compose naturally. + +```ts +// Before - one facade call per service +;async (c) => { + await SessionRunState.assertNotBusy(id) + await Session.removeMessage({ sessionID: id, messageID }) + return c.json(true) +} + +// After - one Effect.gen, yield services from context +;async (c) => { + await AppRuntime.runPromise( + Effect.gen(function* () { + const state = yield* SessionRunState.Service + const session = yield* Session.Service + yield* state.assertNotBusy(id) + yield* session.removeMessage({ sessionID: id, messageID }) + }), + ) + return c.json(true) +} +``` + +## Rules + +- Wrap the whole handler body in one `AppRuntime.runPromise(Effect.gen(...))` call when the handler is service-heavy. +- Yield services from context instead of calling async facades repeatedly. +- When independent service calls can run in parallel, use `Effect.all(..., { concurrency: "unbounded" })`. +- Prefer one composed Effect body over multiple separate `runPromise(...)` calls in the same handler. + +## Current route files + +Current instance route files live under `src/server/instance`, not `server/routes`. + +The main migration targets are: + +- [ ] `server/instance/session.ts` — heaviest; still has many direct facade calls for Session, SessionPrompt, SessionRevert, SessionCompaction, SessionShare, SessionSummary, Agent, Bus +- [ ] `server/instance/global.ts` — still has direct facade calls for Config and instance lifecycle actions +- [ ] `server/instance/provider.ts` — still has direct facade calls for Config and Provider +- [ ] `server/instance/question.ts` — partially converted; still worth tracking here until it consistently uses the composed style +- [ ] `server/instance/pty.ts` — still calls Pty facades directly +- [ ] `server/instance/experimental.ts` — mixed state; some handlers are already composed, others still use facades + +Additional route files that still participate in the migration: + +- [ ] `server/instance/index.ts` — Vcs, Agent, Skill, LSP, Format +- [ ] `server/instance/file.ts` — Ripgrep, File, LSP +- [ ] `server/instance/mcp.ts` — MCP facade-heavy +- [ ] `server/instance/permission.ts` — Permission +- [ ] `server/instance/workspace.ts` — Workspace +- [ ] `server/instance/tui.ts` — Bus and Session +- [ ] `server/instance/middleware.ts` — Session and Workspace lookups + +## Notes + +- Some handlers already use `AppRuntime.runPromise(Effect.gen(...))` in isolated places. Keep pushing those files toward one consistent style. +- Route conversion is closely tied to facade removal. As services lose `makeRuntime`-backed async exports, route handlers should switch to yielding the service directly. diff --git a/packages/opencode/specs/effect/schema.md b/packages/opencode/specs/effect/schema.md new file mode 100644 index 000000000..eed69e52b --- /dev/null +++ b/packages/opencode/specs/effect/schema.md @@ -0,0 +1,99 @@ +# Schema migration + +Practical reference for migrating data types in `packages/opencode` from Zod-first definitions to Effect Schema with Zod compatibility shims. + +## Goal + +Use Effect Schema as the source of truth for domain models, IDs, inputs, outputs, and typed errors. + +Keep Zod available at existing HTTP, tool, and compatibility boundaries by exposing a `.zod` field derived from the Effect schema. + +## Preferred shapes + +### Data objects + +Use `Schema.Class` for structured data. + +```ts +export class Info extends Schema.Class<Info>("Foo.Info")({ + id: FooID, + name: Schema.String, + enabled: Schema.Boolean, +}) { + static readonly zod = zod(Info) +} +``` + +If the class cannot reference itself cleanly during initialization, use the existing two-step pattern: + +```ts +const _Info = Schema.Struct({ + id: FooID, + name: Schema.String, +}) + +export const Info = Object.assign(_Info, { + zod: zod(_Info), +}) +``` + +### Errors + +Use `Schema.TaggedErrorClass` for domain errors. + +```ts +export class NotFoundError extends Schema.TaggedErrorClass<NotFoundError>()("FooNotFoundError", { + id: FooID, +}) {} +``` + +### IDs and branded leaf types + +Keep branded/schema-backed IDs as Effect schemas and expose `static readonly zod` for compatibility when callers still expect Zod. + +## Compatibility rule + +During migration, route validators, tool parameters, and any existing Zod-based boundary should consume the derived `.zod` schema instead of maintaining a second hand-written Zod schema. + +The default should be: + +- Effect Schema owns the type +- `.zod` exists only as a compatibility surface +- new domain models should not start Zod-first unless there is a concrete boundary-specific need + +## When Zod can stay + +It is fine to keep a Zod-native schema temporarily when: + +- the type is only used at an HTTP or tool boundary +- the validator depends on Zod-only transforms or behavior not yet covered by `zod()` +- the migration would force unrelated churn across a large call graph + +When this happens, prefer leaving a short note or TODO rather than silently creating a parallel schema source of truth. + +## Ordering + +Migrate in this order: + +1. Shared leaf models and `schema.ts` files +2. Exported `Info`, `Input`, `Output`, and DTO types +3. Tagged domain errors +4. Service-local internal models +5. Route and tool boundary validators that can switch to `.zod` + +This keeps shared types canonical first and makes boundary updates mostly mechanical. + +## Checklist + +- [ ] Shared `schema.ts` leaf models are Effect Schema-first +- [ ] Exported `Info` / `Input` / `Output` types use `Schema.Class` where appropriate +- [ ] Domain errors use `Schema.TaggedErrorClass` +- [ ] Migrated types expose `.zod` for back compatibility +- [ ] Route and tool validators consume derived `.zod` instead of duplicate Zod definitions +- [ ] New domain models default to Effect Schema first + +## Notes + +- Use `@/util/effect-zod` for all Schema -> Zod conversion. +- Prefer one canonical schema definition. Avoid maintaining parallel Zod and Effect definitions for the same domain type. +- Keep the migration incremental. Converting the domain model first is more valuable than converting every boundary in the same change. diff --git a/packages/opencode/specs/effect/tools.md b/packages/opencode/specs/effect/tools.md new file mode 100644 index 000000000..e97e0d23e --- /dev/null +++ b/packages/opencode/specs/effect/tools.md @@ -0,0 +1,96 @@ +# Tool migration + +Practical reference for the current tool-migration state in `packages/opencode`. + +## Status + +`Tool.Def.execute` and `Tool.Info.init` already return `Effect` on this branch, and the built-in tool surface is now largely on the target shape. + +The current exported tools in `src/tool` all use `Tool.define(...)` with Effect-based initialization, and nearly all of them already build their tool body with `Effect.gen(...)` and `Effect.fn(...)`. + +So the remaining work is no longer "convert tools to Effect at all". The remaining work is mostly: + +1. remove Promise and raw platform bridges inside individual tool bodies +2. swap tool internals to Effect-native services like `AppFileSystem`, `HttpClient`, and `ChildProcessSpawner` +3. keep tests and callers aligned with `yield* info.init()` and real service graphs + +## Current shape + +`Tool.define(...)` is already the Effect-native helper here. + +- `init` is an `Effect` +- `info.init()` returns an `Effect` +- `execute(...)` returns an `Effect` + +That means a tool does not need a separate `Tool.defineEffect(...)` helper to count as migrated. A tool is effectively migrated when its init and execute path stay Effect-native, even if some internals still bridge to Promise-based or raw APIs. + +## Tests + +Tool tests should use the existing Effect helpers in `packages/opencode/test/lib/effect.ts`: + +- Use `testEffect(...)` / `it.live(...)` instead of creating fake local wrappers around effectful tools. +- Yield the real tool export, then initialize it: `const info = yield* ReadTool`, `const tool = yield* info.init()`. +- Run tests inside a real instance with `provideTmpdirInstance(...)` or `provideInstance(tmpdirScoped(...))` so instance-scoped services resolve exactly as they do in production. + +This keeps tool tests aligned with the production service graph and makes follow-up cleanup mostly mechanical. + +## Exported tools + +These exported tool definitions already exist in `src/tool` and are on the current Effect-native `Tool.define(...)` path: + +- [x] `apply_patch.ts` +- [x] `bash.ts` +- [x] `codesearch.ts` +- [x] `edit.ts` +- [x] `glob.ts` +- [x] `grep.ts` +- [x] `invalid.ts` +- [x] `ls.ts` +- [x] `lsp.ts` +- [x] `multiedit.ts` +- [x] `plan.ts` +- [x] `question.ts` +- [x] `read.ts` +- [x] `skill.ts` +- [x] `task.ts` +- [x] `todo.ts` +- [x] `webfetch.ts` +- [x] `websearch.ts` +- [x] `write.ts` + +Notes: + +- `batch.ts` is no longer a current tool file and should not be tracked here. +- `truncate.ts` is an Effect service used by tools, not a tool definition itself. +- `mcp-exa.ts`, `external-directory.ts`, and `schema.ts` are support modules, not standalone tool definitions. + +## Follow-up cleanup + +Most exported tools are already on the intended Effect-native shape. The remaining cleanup is narrower than the old checklist implied. + +Current spot cleanups worth tracking: + +- [ ] `read.ts` — still bridges to Node stream / `readline` helpers and Promise-based binary detection +- [ ] `bash.ts` — already uses Effect child-process primitives; only keep tracking shell-specific platform bridges and parser/loading details as they come up +- [ ] `webfetch.ts` — already uses `HttpClient`; remaining work is limited to smaller boundary helpers like HTML text extraction +- [ ] `file/ripgrep.ts` — adjacent to tool migration; still has raw fs/process usage that affects `grep.ts` and `ls.ts` +- [ ] `patch/index.ts` — adjacent to tool migration; still has raw fs usage behind patch application + +Notable items that are already effectively on the target path and do not need separate migration bullets right now: + +- `apply_patch.ts` +- `grep.ts` +- `write.ts` +- `codesearch.ts` +- `websearch.ts` +- `ls.ts` +- `multiedit.ts` +- `edit.ts` + +## Filesystem notes + +Current raw fs users that still appear relevant here: + +- `tool/read.ts` — `fs.createReadStream`, `readline` +- `file/ripgrep.ts` — `fs/promises` +- `patch/index.ts` — `fs`, `fs/promises` diff --git a/packages/opencode/src/cli/cmd/debug/ripgrep.ts b/packages/opencode/src/cli/cmd/debug/ripgrep.ts index a4cebc5b8..d69348b30 100644 --- a/packages/opencode/src/cli/cmd/debug/ripgrep.ts +++ b/packages/opencode/src/cli/cmd/debug/ripgrep.ts @@ -1,4 +1,5 @@ import { EOL } from "os" +import { AppRuntime } from "../../../effect/app-runtime" import { Ripgrep } from "../../../file/ripgrep" import { Instance } from "../../../project/instance" import { bootstrap } from "../../bootstrap" @@ -76,12 +77,18 @@ const SearchCommand = cmd({ description: "Limit number of results", }), async handler(args) { - const results = await Ripgrep.search({ - cwd: process.cwd(), - pattern: args.pattern, - glob: args.glob as string[] | undefined, - limit: args.limit, + await bootstrap(process.cwd(), async () => { + const results = await AppRuntime.runPromise( + Ripgrep.Service.use((svc) => + svc.search({ + cwd: Instance.directory, + pattern: args.pattern, + glob: args.glob as string[] | undefined, + limit: args.limit, + }), + ), + ) + process.stdout.write(JSON.stringify(results.items, null, 2) + EOL) }) - process.stdout.write(JSON.stringify(results, null, 2) + EOL) }, }) diff --git a/packages/opencode/src/file/ripgrep.ts b/packages/opencode/src/file/ripgrep.ts index 4d0fc5598..4100563d5 100644 --- a/packages/opencode/src/file/ripgrep.ts +++ b/packages/opencode/src/file/ripgrep.ts @@ -96,6 +96,7 @@ export namespace Ripgrep { export type Result = z.infer<typeof Result> export type Match = z.infer<typeof Match> + export type Item = Match["data"] export type Begin = z.infer<typeof Begin> export type End = z.infer<typeof End> export type Summary = z.infer<typeof Summary> @@ -289,6 +290,13 @@ export namespace Ripgrep { follow?: boolean maxDepth?: number }) => Stream.Stream<string, PlatformError> + readonly search: (input: { + cwd: string + pattern: string + glob?: string[] + limit?: number + follow?: boolean + }) => Effect.Effect<{ items: Item[]; partial: boolean }, PlatformError | Error> } export class Service extends Context.Service<Service, Interface>()("@opencode/Ripgrep") {} @@ -298,6 +306,32 @@ export namespace Ripgrep { Effect.gen(function* () { const spawner = yield* ChildProcessSpawner const afs = yield* AppFileSystem.Service + const bin = Effect.fn("Ripgrep.path")(function* () { + return yield* Effect.promise(() => filepath()) + }) + const args = Effect.fn("Ripgrep.args")(function* (input: { + mode: "files" | "search" + glob?: string[] + hidden?: boolean + follow?: boolean + maxDepth?: number + limit?: number + pattern?: string + }) { + const out = [yield* bin(), input.mode === "search" ? "--json" : "--files", "--glob=!.git/*"] + if (input.follow) out.push("--follow") + if (input.hidden !== false) out.push("--hidden") + if (input.maxDepth !== undefined) out.push(`--max-depth=${input.maxDepth}`) + if (input.glob) { + for (const g of input.glob) { + out.push(`--glob=${g}`) + } + } + if (input.limit) out.push(`--max-count=${input.limit}`) + if (input.mode === "search") out.push("--no-messages") + if (input.pattern) out.push("--", input.pattern) + return out + }) const files = Effect.fn("Ripgrep.files")(function* (input: { cwd: string @@ -306,7 +340,7 @@ export namespace Ripgrep { follow?: boolean maxDepth?: number }) { - const rgPath = yield* Effect.promise(() => filepath()) + const rgPath = yield* bin() const isDir = yield* afs.isDir(input.cwd) if (!isDir) { return yield* Effect.die( @@ -318,23 +352,76 @@ export namespace Ripgrep { ) } - const args = [rgPath, "--files", "--glob=!.git/*"] - if (input.follow) args.push("--follow") - if (input.hidden !== false) args.push("--hidden") - if (input.maxDepth !== undefined) args.push(`--max-depth=${input.maxDepth}`) - if (input.glob) { - for (const g of input.glob) { - args.push(`--glob=${g}`) - } - } + const cmd = yield* args({ + mode: "files", + glob: input.glob, + hidden: input.hidden, + follow: input.follow, + maxDepth: input.maxDepth, + }) return spawner - .streamLines(ChildProcess.make(args[0], args.slice(1), { cwd: input.cwd })) + .streamLines(ChildProcess.make(cmd[0], cmd.slice(1), { cwd: input.cwd })) .pipe(Stream.filter((line: string) => line.length > 0)) }) + const search = Effect.fn("Ripgrep.search")(function* (input: { + cwd: string + pattern: string + glob?: string[] + limit?: number + follow?: boolean + }) { + return yield* Effect.scoped( + Effect.gen(function* () { + const cmd = yield* args({ + mode: "search", + glob: input.glob, + follow: input.follow, + limit: input.limit, + pattern: input.pattern, + }) + + const handle = yield* spawner.spawn( + ChildProcess.make(cmd[0], cmd.slice(1), { + cwd: input.cwd, + stdin: "ignore", + }), + ) + + const [stdout, stderr, code] = yield* Effect.all( + [ + Stream.mkString(Stream.decodeText(handle.stdout)), + Stream.mkString(Stream.decodeText(handle.stderr)), + handle.exitCode, + ], + { concurrency: "unbounded" }, + ) + + if (code !== 0 && code !== 1 && code !== 2) { + return yield* Effect.fail(new Error(`ripgrep failed: ${stderr}`)) + } + + const items = stdout + .trim() + .split(/\r?\n/) + .filter(Boolean) + .map((line) => JSON.parse(line)) + .map((parsed) => Result.parse(parsed)) + .filter((row): row is Match => row.type === "match") + .map((row) => row.data) + + return { + items, + partial: code === 2, + } + }), + ) + }) + return Service.of({ files: (input) => Stream.unwrap(files(input)), + search, }) }), ) @@ -401,46 +488,4 @@ export namespace Ripgrep { return lines.join("\n") } - - export async function search(input: { - cwd: string - pattern: string - glob?: string[] - limit?: number - follow?: boolean - }) { - const args = [`${await filepath()}`, "--json", "--hidden", "--glob=!.git/*"] - if (input.follow) args.push("--follow") - - if (input.glob) { - for (const g of input.glob) { - args.push(`--glob=${g}`) - } - } - - if (input.limit) { - args.push(`--max-count=${input.limit}`) - } - - args.push("--") - args.push(input.pattern) - - const result = await Process.text(args, { - cwd: input.cwd, - nothrow: true, - }) - if (result.code !== 0) { - return [] - } - - // Handle both Unix (\n) and Windows (\r\n) line endings - const lines = result.text.trim().split(/\r?\n/).filter(Boolean) - // Parse JSON lines from ripgrep output - - return lines - .map((line) => JSON.parse(line)) - .map((parsed) => Result.parse(parsed)) - .filter((r) => r.type === "match") - .map((r) => r.data) - } } diff --git a/packages/opencode/src/server/instance/file.ts b/packages/opencode/src/server/instance/file.ts index 60789ef4b..713513b38 100644 --- a/packages/opencode/src/server/instance/file.ts +++ b/packages/opencode/src/server/instance/file.ts @@ -1,6 +1,7 @@ import { Hono } from "hono" import { describeRoute, validator, resolver } from "hono-openapi" import z from "zod" +import { AppRuntime } from "../../effect/app-runtime" import { File } from "../../file" import { Ripgrep } from "../../file/ripgrep" import { LSP } from "../../lsp" @@ -34,12 +35,10 @@ export const FileRoutes = lazy(() => ), async (c) => { const pattern = c.req.valid("query").pattern - const result = await Ripgrep.search({ - cwd: Instance.directory, - pattern, - limit: 10, - }) - return c.json(result) + const result = await AppRuntime.runPromise( + Ripgrep.Service.use((svc) => svc.search({ cwd: Instance.directory, pattern, limit: 10 })), + ) + return c.json(result.items) }, ) .get( diff --git a/packages/opencode/src/tool/grep.ts b/packages/opencode/src/tool/grep.ts index b5ae6c350..9b5143cec 100644 --- a/packages/opencode/src/tool/grep.ts +++ b/packages/opencode/src/tool/grep.ts @@ -1,11 +1,8 @@ import z from "zod" -import { Effect } from "effect" -import * as Stream from "effect/Stream" +import { Effect, Option } from "effect" import { Tool } from "./tool" -import { Filesystem } from "../util/filesystem" import { Ripgrep } from "../file/ripgrep" -import { ChildProcess } from "effect/unstable/process" -import { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner" +import { AppFileSystem } from "../filesystem" import DESCRIPTION from "./grep.txt" import { Instance } from "../project/instance" @@ -17,7 +14,8 @@ const MAX_LINE_LENGTH = 2000 export const GrepTool = Tool.define( "grep", Effect.gen(function* () { - const spawner = yield* ChildProcessSpawner + const fs = yield* AppFileSystem.Service + const rg = yield* Ripgrep.Service return { description: DESCRIPTION, @@ -28,6 +26,11 @@ export const GrepTool = Tool.define( }), execute: (params: { pattern: string; path?: string; include?: string }, ctx: Tool.Context) => Effect.gen(function* () { + const empty = { + title: params.pattern, + metadata: { matches: 0, truncated: false }, + output: "No files found", + } if (!params.pattern) { throw new Error("pattern is required") } @@ -43,92 +46,58 @@ export const GrepTool = Tool.define( }, }) - let searchPath = params.path ?? Instance.directory - searchPath = path.isAbsolute(searchPath) ? searchPath : path.resolve(Instance.directory, searchPath) + const searchPath = AppFileSystem.resolve( + path.isAbsolute(params.path ?? Instance.directory) + ? (params.path ?? Instance.directory) + : path.join(Instance.directory, params.path ?? "."), + ) yield* assertExternalDirectoryEffect(ctx, searchPath, { kind: "directory" }) - const rgPath = yield* Effect.promise(() => Ripgrep.filepath()) - const args = ["-nH", "--hidden", "--no-messages", "--field-match-separator=|", "--regexp", params.pattern] - if (params.include) { - args.push("--glob", params.include) - } - args.push(searchPath) - - const result = yield* Effect.scoped( - Effect.gen(function* () { - const handle = yield* spawner.spawn( - ChildProcess.make(rgPath, args, { - stdin: "ignore", - }), - ) - - const [output, errorOutput] = yield* Effect.all( - [Stream.mkString(Stream.decodeText(handle.stdout)), Stream.mkString(Stream.decodeText(handle.stderr))], - { concurrency: 2 }, - ) - - const exitCode = yield* handle.exitCode + const result = yield* rg.search({ + cwd: searchPath, + pattern: params.pattern, + glob: params.include ? [params.include] : undefined, + }) - return { output, errorOutput, exitCode } - }), + if (result.items.length === 0) return empty + + const rows = result.items.map((item) => ({ + path: AppFileSystem.resolve( + path.isAbsolute(item.path.text) ? item.path.text : path.join(searchPath, item.path.text), + ), + line: item.line_number, + text: item.lines.text, + })) + const times = new Map( + (yield* Effect.forEach( + [...new Set(rows.map((row) => row.path))], + Effect.fnUntraced(function* (file) { + const info = yield* fs.stat(file).pipe(Effect.catch(() => Effect.succeed(undefined))) + if (!info || info.type === "Directory") return undefined + return [ + file, + info.mtime.pipe( + Option.map((time) => time.getTime()), + Option.getOrElse(() => 0), + ) ?? 0, + ] as const + }), + { concurrency: 16 }, + )).filter((entry): entry is readonly [string, number] => Boolean(entry)), ) + const matches = rows.flatMap((row) => { + const mtime = times.get(row.path) + if (mtime === undefined) return [] + return [{ ...row, mtime }] + }) - const { output, errorOutput, exitCode } = result - - // Exit codes: 0 = matches found, 1 = no matches, 2 = errors (but may still have matches) - // With --no-messages, we suppress error output but still get exit code 2 for broken symlinks etc. - // Only fail if exit code is 2 AND no output was produced - if (exitCode === 1 || (exitCode === 2 && !output.trim())) { - return { - title: params.pattern, - metadata: { matches: 0, truncated: false }, - output: "No files found", - } - } - - if (exitCode !== 0 && exitCode !== 2) { - throw new Error(`ripgrep failed: ${errorOutput}`) - } - - const hasErrors = exitCode === 2 - - // Handle both Unix (\n) and Windows (\r\n) line endings - const lines = output.trim().split(/\r?\n/) - const matches = [] - - for (const line of lines) { - if (!line) continue - - const [filePath, lineNumStr, ...lineTextParts] = line.split("|") - if (!filePath || !lineNumStr || lineTextParts.length === 0) continue - - const lineNum = parseInt(lineNumStr, 10) - const lineText = lineTextParts.join("|") - - const stats = Filesystem.stat(filePath) - if (!stats) continue - - matches.push({ - path: filePath, - modTime: stats.mtime.getTime(), - lineNum, - lineText, - }) - } - - matches.sort((a, b) => b.modTime - a.modTime) + matches.sort((a, b) => b.mtime - a.mtime) const limit = 100 const truncated = matches.length > limit const finalMatches = truncated ? matches.slice(0, limit) : matches - if (finalMatches.length === 0) { - return { - title: params.pattern, - metadata: { matches: 0, truncated: false }, - output: "No files found", - } - } + if (finalMatches.length === 0) return empty const totalMatches = matches.length const outputLines = [`Found ${totalMatches} matches${truncated ? ` (showing first ${limit})` : ""}`] @@ -143,10 +112,8 @@ export const GrepTool = Tool.define( outputLines.push(`${match.path}:`) } const truncatedLineText = - match.lineText.length > MAX_LINE_LENGTH - ? match.lineText.substring(0, MAX_LINE_LENGTH) + "..." - : match.lineText - outputLines.push(` Line ${match.lineNum}: ${truncatedLineText}`) + match.text.length > MAX_LINE_LENGTH ? match.text.substring(0, MAX_LINE_LENGTH) + "..." : match.text + outputLines.push(` Line ${match.line}: ${truncatedLineText}`) } if (truncated) { @@ -156,7 +123,7 @@ export const GrepTool = Tool.define( ) } - if (hasErrors) { + if (result.partial) { outputLines.push("") outputLines.push("(Some paths were inaccessible and skipped)") } diff --git a/packages/opencode/test/file/ripgrep.test.ts b/packages/opencode/test/file/ripgrep.test.ts index 3982c184f..11d212a08 100644 --- a/packages/opencode/test/file/ripgrep.test.ts +++ b/packages/opencode/test/file/ripgrep.test.ts @@ -38,7 +38,9 @@ describe("file.ripgrep", () => { expect(hasVisible).toBe(true) expect(hasHidden).toBe(false) }) +}) +describe("Ripgrep.Service", () => { test("search returns empty when nothing matches", async () => { await using tmp = await tmpdir({ init: async (dir) => { @@ -46,16 +48,34 @@ describe("file.ripgrep", () => { }, }) - const hits = await Ripgrep.search({ - cwd: tmp.path, - pattern: "needle", + const result = await Effect.gen(function* () { + const rg = yield* Ripgrep.Service + return yield* rg.search({ cwd: tmp.path, pattern: "needle" }) + }).pipe(Effect.provide(Ripgrep.defaultLayer), Effect.runPromise) + + expect(result.partial).toBe(false) + expect(result.items).toEqual([]) + }) + + test("search returns matched rows", async () => { + await using tmp = await tmpdir({ + init: async (dir) => { + await Bun.write(path.join(dir, "match.ts"), "const value = 'needle'\n") + await Bun.write(path.join(dir, "skip.txt"), "const value = 'other'\n") + }, }) - expect(hits).toEqual([]) + const result = await Effect.gen(function* () { + const rg = yield* Ripgrep.Service + return yield* rg.search({ cwd: tmp.path, pattern: "needle", glob: ["*.ts"] }) + }).pipe(Effect.provide(Ripgrep.defaultLayer), Effect.runPromise) + + expect(result.partial).toBe(false) + expect(result.items).toHaveLength(1) + expect(result.items[0]?.path.text).toContain("match.ts") + expect(result.items[0]?.lines.text).toContain("needle") }) -}) -describe("Ripgrep.Service", () => { test("files returns stream of filenames", async () => { await using tmp = await tmpdir({ init: async (dir) => { diff --git a/packages/opencode/test/tool/grep.test.ts b/packages/opencode/test/tool/grep.test.ts index 4715ad925..07ac231df 100644 --- a/packages/opencode/test/tool/grep.test.ts +++ b/packages/opencode/test/tool/grep.test.ts @@ -1,22 +1,26 @@ -import { describe, expect, test } from "bun:test" +import { describe, expect } from "bun:test" import path from "path" -import { Effect, Layer, ManagedRuntime } from "effect" +import { Effect, Layer } from "effect" import { GrepTool } from "../../src/tool/grep" -import { Instance } from "../../src/project/instance" -import { tmpdir } from "../fixture/fixture" +import { provideInstance, provideTmpdirInstance } from "../fixture/fixture" import { SessionID, MessageID } from "../../src/session/schema" import * as CrossSpawnSpawner from "../../src/effect/cross-spawn-spawner" import { Truncate } from "../../src/tool/truncate" import { Agent } from "../../src/agent/agent" +import { Ripgrep } from "../../src/file/ripgrep" +import { AppFileSystem } from "../../src/filesystem" +import { testEffect } from "../lib/effect" -const runtime = ManagedRuntime.make( - Layer.mergeAll(CrossSpawnSpawner.defaultLayer, Truncate.defaultLayer, Agent.defaultLayer), +const it = testEffect( + Layer.mergeAll( + CrossSpawnSpawner.defaultLayer, + AppFileSystem.defaultLayer, + Ripgrep.defaultLayer, + Truncate.defaultLayer, + Agent.defaultLayer, + ), ) -function initGrep() { - return runtime.runPromise(GrepTool.pipe(Effect.flatMap((info) => info.init()))) -} - const ctx = { sessionID: SessionID.make("ses_test"), messageID: MessageID.make(""), @@ -31,99 +35,59 @@ const ctx = { const projectRoot = path.join(__dirname, "../..") describe("tool.grep", () => { - test("basic search", async () => { - await Instance.provide({ - directory: projectRoot, - fn: async () => { - const grep = await initGrep() - const result = await Effect.runPromise( - grep.execute( - { - pattern: "export", - path: path.join(projectRoot, "src/tool"), - include: "*.ts", - }, - ctx, - ), - ) - expect(result.metadata.matches).toBeGreaterThan(0) - expect(result.output).toContain("Found") - }, - }) - }) + it.live("basic search", () => + Effect.gen(function* () { + const info = yield* GrepTool + const grep = yield* info.init() + const result = yield* provideInstance(projectRoot)( + grep.execute( + { + pattern: "export", + path: path.join(projectRoot, "src/tool"), + include: "*.ts", + }, + ctx, + ), + ) + expect(result.metadata.matches).toBeGreaterThan(0) + expect(result.output).toContain("Found") + }), + ) - test("no matches returns correct output", async () => { - await using tmp = await tmpdir({ - init: async (dir) => { - await Bun.write(path.join(dir, "test.txt"), "hello world") - }, - }) - await Instance.provide({ - directory: tmp.path, - fn: async () => { - const grep = await initGrep() - const result = await Effect.runPromise( - grep.execute( - { - pattern: "xyznonexistentpatternxyz123", - path: tmp.path, - }, - ctx, - ), + it.live("no matches returns correct output", () => + provideTmpdirInstance((dir) => + Effect.gen(function* () { + yield* Effect.promise(() => Bun.write(path.join(dir, "test.txt"), "hello world")) + const info = yield* GrepTool + const grep = yield* info.init() + const result = yield* grep.execute( + { + pattern: "xyznonexistentpatternxyz123", + path: dir, + }, + ctx, ) expect(result.metadata.matches).toBe(0) expect(result.output).toBe("No files found") - }, - }) - }) + }), + ), + ) - test("handles CRLF line endings in output", async () => { - // This test verifies the regex split handles both \n and \r\n - await using tmp = await tmpdir({ - init: async (dir) => { - // Create a test file with content - await Bun.write(path.join(dir, "test.txt"), "line1\nline2\nline3") - }, - }) - await Instance.provide({ - directory: tmp.path, - fn: async () => { - const grep = await initGrep() - const result = await Effect.runPromise( - grep.execute( - { - pattern: "line", - path: tmp.path, - }, - ctx, - ), + it.live("finds matches in tmp instance", () => + provideTmpdirInstance((dir) => + Effect.gen(function* () { + yield* Effect.promise(() => Bun.write(path.join(dir, "test.txt"), "line1\nline2\nline3")) + const info = yield* GrepTool + const grep = yield* info.init() + const result = yield* grep.execute( + { + pattern: "line", + path: dir, + }, + ctx, ) expect(result.metadata.matches).toBeGreaterThan(0) - }, - }) - }) -}) - -describe("CRLF regex handling", () => { - test("regex correctly splits Unix line endings", () => { - const unixOutput = "file1.txt|1|content1\nfile2.txt|2|content2\nfile3.txt|3|content3" - const lines = unixOutput.trim().split(/\r?\n/) - expect(lines.length).toBe(3) - expect(lines[0]).toBe("file1.txt|1|content1") - expect(lines[2]).toBe("file3.txt|3|content3") - }) - - test("regex correctly splits Windows CRLF line endings", () => { - const windowsOutput = "file1.txt|1|content1\r\nfile2.txt|2|content2\r\nfile3.txt|3|content3" - const lines = windowsOutput.trim().split(/\r?\n/) - expect(lines.length).toBe(3) - expect(lines[0]).toBe("file1.txt|1|content1") - expect(lines[2]).toBe("file3.txt|3|content3") - }) - - test("regex handles mixed line endings", () => { - const mixedOutput = "file1.txt|1|content1\nfile2.txt|2|content2\r\nfile3.txt|3|content3" - const lines = mixedOutput.trim().split(/\r?\n/) - expect(lines.length).toBe(3) - }) + }), + ), + ) }) |