diff options
Diffstat (limited to 'cc')
| -rw-r--r-- | cc/01-tool-schema-analysis.md | 62 | ||||
| -rw-r--r-- | cc/02-anthropic-tool-format.md | 90 | ||||
| -rw-r--r-- | cc/03-recommendations.md | 116 | ||||
| -rw-r--r-- | cc/04-schema-debug.md | 40 | ||||
| -rw-r--r-- | cc/README.md | 38 |
5 files changed, 346 insertions, 0 deletions
diff --git a/cc/01-tool-schema-analysis.md b/cc/01-tool-schema-analysis.md new file mode 100644 index 0000000..d803b2c --- /dev/null +++ b/cc/01-tool-schema-analysis.md @@ -0,0 +1,62 @@ +# Tool Schema Analysis — Dispatch / AI SDK v6 + +## How Dispatch Defines Tools + +Dispatch uses the **AI SDK v6** (`ai@^6.0.191`) with provider adapters (`@ai-sdk/anthropic@^3.0.79`, `@ai-sdk/openai-compatible@^2.0.48`). + +### The conversion pipeline + +``` +Zod schema (z.object({...})) + → zodToJsonSchema() → JSON Schema Draft 7 + → jsonSchema() → AI SDK v6 inputSchema wrapper + → tool() → AI SDK v6 Tool object (no execute fn) + → streamText({tools}) → SDK → provider adapter → wire format +``` + +**Critical file:** `packages/core/src/tools/registry.ts` — the `toAISDKTool()` function. + +### What each tool's schema looks like after conversion + +For a typical tool like `read_file`: +```json +{ + "type": "object", + "properties": { + "path": { "type": "string", "description": "Path to the file..." }, + "offset": { "type": "integer", "minimum": 1, "description": "..." }, + "limit": { "type": "integer", "minimum": 1, "description": "..." } + }, + "required": ["path"], + "additionalProperties": false, + "$schema": "http://json-schema.org/draft-07/schema#" +} +``` + +### The Problem Surface + +The `zodToJsonSchema()` library (v3.x) generates full Draft 7 JSON Schema including: + +1. **`$schema` field** — `"http://json-schema.org/draft-07/schema#"` — Anthropic's API rejects this or silently ignores it depending on version +2. **`additionalProperties: false`** — Anthropic may not handle this correctly; Claude's `tool_use` blocks sometimes include extra fields +3. **`default` values** — Zod schemas with `.default()` produce JSON Schema `default` fields that Anthropic doesn't support +4. **`minimum`/`maximum`** — Numeric constraints may cause issues if Claude passes values slightly outside bounds + +### What the oh-my-pi reference does differently + +The `references/oh-my-pi/packages/ai/src/utils/schema/normalize.ts` shows a `normalizeSchemaForCCA()` function that strips all of the above. Dispatch does **no normalization** before passing schemas to the AI SDK. + +### What the AI SDK's @ai-sdk/anthropic adapter does + +The adapter should convert the AI SDK's internal format to Anthropic's wire format. However, the adapter may: +- Pass through JSON Schema fields that Anthropic doesn't support (like `$schema`, `additionalProperties`) +- Not add parameter `description` fields from the schema to the Anthropic format +- Not handle `default` values correctly + +### Files to inspect + +| File | What to check | +|------|---------------| +| `node_modules/@ai-sdk/anthropic/dist/index.mjs` | How tools are serialized for the wire | +| `node_modules/ai/dist/index.mjs` | How `tool()` and `jsonSchema()` work | +| `node_modules/zod-to-json-schema/dist/` | What `zodToJsonSchema` outputs for our schemas | diff --git a/cc/02-anthropic-tool-format.md b/cc/02-anthropic-tool-format.md new file mode 100644 index 0000000..3b2888c --- /dev/null +++ b/cc/02-anthropic-tool-format.md @@ -0,0 +1,90 @@ +# Anthropic Tool Format — What Claude Actually Expects + +## The Correct Anthropic API Tool Format + +```json +{ + "tools": [ + { + "name": "tool_name", + "description": "What this tool does", + "input_schema": { + "type": "object", + "properties": { + "param1": { + "type": "string", + "description": "What param1 is" + } + }, + "required": ["param1"] + } + } + ], + "tool_choice": { "type": "auto" } +} +``` + +### Key requirements + +| Field | Type | Required | Notes | +|-------|------|----------|-------| +| `name` | string | **Yes** | Must match `^[a-zA-Z0-9_-]{1,64}$` | +| `description` | string | Strongly recommended | Claude uses this to decide when to call | +| `input_schema` | object | **Yes** | JSON Schema; root must have `type: "object"` | +| `input_schema.type` | string | **Yes** | MUST be `"object"` | +| `input_schema.properties` | object | Recommended | Each property needs `type` and `description` | +| `input_schema.required` | string[] | Optional | Lists required property names | + +### Anthropic does NOT support in input_schema + +1. **`$schema`** — JSON Schema meta-keyword; Anthropic rejects it +2. **`additionalProperties`** — Not supported; can cause silent schema rejection +3. **`default`** — Not part of Anthropic's JSON Schema subset +4. **`type` as array** (e.g. `["string", "null"]`) — Rejected +5. **`type: "null"`** — Rejected +6. **`nullable`** keyword — Not supported +7. **`anyOf` / `oneOf` / `allOf`** — Not supported (combiners) +8. **`$ref` / `$defs` / `$dynamicRef`** — Not supported +9. **`propertyNames`** — Not supported + +## What Claude Code (the CLI) Uses Internally + +Claude Code has its OWN internal tool definitions. They use a specific format that Claude has been trained on extensively. When you use Claude outside of Claude Code, the model loses: + +1. **The exact tool descriptions** it was trained on in Claude Code +2. **The precise schema format** Claude Code uses +3. **The system prompt** that tells Claude to use tools proactively +4. **The billing/identity headers** that tell Anthropic this is a "real" CLI session + +### The billing header trick + +From `packages/core/src/credentials/claude.ts`: + +``` +x-anthropic-billing-header: cc_version=2.1.112.xxx; cc_entrypoint=sdk-cli; cch=xxxxx; +``` + +And the identity preamble: +``` +"You are Claude Code, Anthropic's official CLI for Claude." +``` + +These are mirrored from opencode, but **they go beyond what the raw API needs**. They're hacks to get Anthropic's backend to treat the request as coming from Claude Code. + +## Why Claude Opus "Thinks Forever" + +Common causes when tools don't get called: + +1. **Tool schema has unsupported fields** → Anthropic silently ignores the tool → Claude never sees it → Claude just talks +2. **Missing description on parameters** → Claude doesn't know what values to provide → stalls +3. **`input_schema` missing `type: "object"`** → Anthropic rejects the tool entirely +4. **`tool_choice` is wrong** → Set to `"none"` or Claude decides not to use tools +5. **System prompt doesn't instruct tool use** → Claude doesn't realize it should call tools +6. **Anthropic beta headers missing** → Extended thinking or new features might not work +7. **Thinking budget too high** → Claude uses all tokens thinking and never gets to tool calls + +## Sources + +- https://docs.anthropic.com/en/docs/build-with-claude/tool-use +- https://docs.aimlapi.com/capabilities/anthropic +- https://sdk.vercel.ai/providers/ai-sdk-providers/anthropic diff --git a/cc/03-recommendations.md b/cc/03-recommendations.md new file mode 100644 index 0000000..47ffaf8 --- /dev/null +++ b/cc/03-recommendations.md @@ -0,0 +1,116 @@ +# Recommendations — Fixing Claude Opus Tool Calling + +## 1. Add Anthropic Schema Normalization + +**Problem:** `zodToJsonSchema()` generates Draft 7 JSON Schema with `$schema`, `additionalProperties`, and potentially `default` fields that Anthropic's API doesn't support. + +**Fix:** Add a normalization step between `zodToJsonSchema()` and `jsonSchema()` in `packages/core/src/tools/registry.ts` that strips unsupported fields: + +```typescript +function normalizeForAnthropic(schema: Record<string, unknown>): Record<string, unknown> { + // Remove fields Anthropic doesn't support + delete schema.$schema; + delete schema.additionalProperties; + delete schema.default; + // Strip from nested properties too + if (schema.properties && typeof schema.properties === 'object') { + for (const key of Object.keys(schema.properties as Record<string, unknown>)) { + const prop = (schema.properties as Record<string, unknown>)[key] as Record<string, unknown>; + delete prop.$schema; + delete prop.additionalProperties; + delete prop.default; + } + } + return schema; +} +``` + +**File to modify:** `packages/core/src/tools/registry.ts` — the `toAISDKTool()` function. + +## 2. Verify @ai-sdk/anthropic Adapter Version + +**Check:** `node_modules/@ai-sdk/anthropic/dist/index.mjs` — verify the adapter properly converts `input_schema` to Anthropic's `input_schema` format on the wire. + +The AI SDK v6 adapter should handle this, but verify by looking at how it serializes tools. The key serialization happens in the adapter's `convertToolsToAnthropic()` or similar function. + +## 3. Add `tool_choice: "any"` for Opus + +**Problem:** Opus may decide not to call tools even when it should. + +**Fix:** For Claude Opus sessions, consider setting `tool_choice: { type: "any" }` (or `"auto"`) to encourage tool use. Currently dispatch doesn't set any explicit `tool_choice` — the AI SDK default may be suboptimal. + +In `packages/core/src/agent/agent.ts`, the `streamText` options don't include `toolChoice`. Consider adding it conditionally for anthropic provider: + +```typescript +const streamOptions = { + model, + messages: coreMessages, + tools, + ...(isClaudeOAuth ? { toolChoice: "auto" } : {}), +}; +``` + +## 4. Check Parameter Descriptions + +**Problem:** All tools have `.describe()` on their parameters, but verify the AI SDK's Anthropic adapter is forwarding these descriptions to the Anthropic `input_schema.properties.*.description` field. + +## 5. System Prompt Tool Instructions + +The current system prompt in `agent-manager.ts` lists tools generically: +``` +"You have access to the following tools:\n\n{tool_list}\n\nWhen asked to work with files, use these tools." +``` + +For Claude Opus, add more explicit instructions about WHEN to use each tool and that it SHOULD use tools rather than just talking about solutions. + +## 6. Debugging: Log the Actual API Request + +Add logging to see what's actually sent to Anthropic for the `tools` parameter. The most reliable way is to add a `fetch` wrapper or check the `@ai-sdk/anthropic` adapter's serialization. + +Quick check: +```bash +node -e " +import { zodToJsonSchema } from 'zod-to-json-schema'; +import { z } from 'zod'; +const schema = z.object({ + path: z.string().describe('Path to the file'), + offset: z.number().int().optional(), +}); +console.log(JSON.stringify(zodToJsonSchema(schema), null, 2)); +" +``` + +This will show exactly what JSON Schema is produced and whether it has `$schema`, `additionalProperties`, etc. + +## 7. Test with Raw Anthropic API + +Bypass the AI SDK entirely and test with a direct API call to isolate whether the issue is in the AI SDK adapter: + +```bash +curl -X POST https://api.anthropic.com/v1/messages \ + -H "anthropic-version: 2023-06-01" \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "content-type: application/json" \ + -d '{ + "model": "claude-opus-4-20250514", + "max_tokens": 1024, + "tools": [ + { + "name": "read_file", + "description": "Read the contents of a file", + "input_schema": { + "type": "object", + "properties": { + "path": {"type": "string", "description": "Path to file"} + }, + "required": ["path"] + } + } + ], + "messages": [ + {"role": "user", "content": "Read /etc/hostname"} + ] + }' +``` + +If this works but dispatch doesn't, the issue is in the AI SDK adapter or the schema conversion. diff --git a/cc/04-schema-debug.md b/cc/04-schema-debug.md new file mode 100644 index 0000000..5889862 --- /dev/null +++ b/cc/04-schema-debug.md @@ -0,0 +1,40 @@ +# Schema Debug — What zodToJsonSchema Actually Produces + +Run this to see what the AI SDK actually sends to Anthropic: + +```bash +node --experimental-strip-types -e " +import { z } from 'zod'; +import { zodToJsonSchema } from 'zod-to-json-schema'; + +const schema = z.object({ + path: z.string().describe('Path to the file, relative to the working directory'), + offset: z.number().int().min(1).optional().describe('1-indexed start line. Default: 1.'), + limit: z.number().int().min(1).optional().describe('Max lines to return. Default: 500. Hard cap: 5000.'), +}); + +console.log(JSON.stringify(zodToJsonSchema(schema), null, 2)); +" +``` + +## Check the @ai-sdk/anthropic adapter's tool serialization + +```bash +grep -n 'input_schema\|tools\|jsonSchema\|convertTools' node_modules/@ai-sdk/anthropic/dist/index.mjs | head -30 +``` + +## Check if streamText is receiving the tools correctly + +Look at how streamText processes tool options in the AI SDK: + +```bash +grep -n 'tools\|toolChoice\|toolCall' node_modules/ai/dist/index.mjs | head -40 +``` + +## Key questions to answer + +1. Does `zodToJsonSchema` output `$schema`? If yes, Anthropic may silently reject the tool definition. +2. Does it output `additionalProperties`? Same concern. +3. Does the `@ai-sdk/anthropic` adapter strip these before sending to the API? +4. Does the adapter forward parameter `description` fields from JSON Schema to Anthropic's wire format? +5. Is `tool_choice` being set or defaulting to something suboptimal? diff --git a/cc/README.md b/cc/README.md new file mode 100644 index 0000000..a9c813d --- /dev/null +++ b/cc/README.md @@ -0,0 +1,38 @@ +# Claude Opus Tool Calling Investigation + +## Problem +Claude Opus "thinks forever" and doesn't call tools when used outside Claude Code's harness (in Dispatch's own agent harness). + +## Summary of Findings + +### 1. Tool Schema Format +Dispatch uses the AI SDK v6 (`@ai-sdk/anthropic@^3.x`) which should handle format conversion automatically. However, `zodToJsonSchema()` produces Draft 7 JSON Schema with fields (`$schema`, `additionalProperties`, `default`) that Anthropic's API doesn't support. Dispatch does **no schema normalization**. + +### 2. Anthropic's requirements +Anthropic's `input_schema` must be clean JSON Schema: +- Root `type: "object"` is mandatory +- No `$schema`, `additionalProperties`, `default`, `nullable` +- No combiners (`anyOf`, `oneOf`, etc.) +- Parameter `description` fields are strongly recommended + +### 3. Missing tool_choice +Dispatch doesn't set `tool_choice` in `streamText()` options. The default may cause Opus to not call tools. + +### 4. System prompt +The system prompt tells Opus what tools exist but may not be forceful enough about actually USING them instead of just talking about solutions. + +### 5. No Anthropic-specific schema normalization +Unlike opencode's `normalizeSchemaForCCA()`, dispatch passes raw JSON Schema to the AI SDK without stripping unsupported fields. + +## Key Files + +| File | Purpose | +|------|---------| +| `packages/core/src/tools/registry.ts` | Tool → AI SDK conversion | +| `packages/core/src/agent/agent.ts` | Agent loop, `streamText({tools})` | +| `packages/api/src/agent-manager.ts` | Provides tool config to agent | +| `packages/core/src/llm/provider.ts` | Provider creation | + +## Next Steps + +See `03-recommendations.md` for specific fixes to try. |