summaryrefslogtreecommitdiffhomepage
path: root/tasks.md
diff options
context:
space:
mode:
authorAdam Malczewski <[email protected]>2026-06-07 17:46:53 +0900
committerAdam Malczewski <[email protected]>2026-06-07 17:46:53 +0900
commit7c459c7d919d1e08a228e8abc56129be174d8abe (patch)
tree93011125c001945723ac9b9358c4ddd450f87f72 /tasks.md
parent5746cf4e545cd5b0d7faf0595554f273f236f3a9 (diff)
downloaddispatch-7c459c7d919d1e08a228e8abc56129be174d8abe.tar.gz
dispatch-7c459c7d919d1e08a228e8abc56129be174d8abe.zip
feat(wire,kernel,session-orchestrator): live turn metrics on the stream
Expose the backend's authoritative token+timing metrics on the live AgentEvent stream (observability-only -> now also client-facing). All additive/optional. - [email protected]: new TurnStepCompleteEvent (type:step-complete) with per-step ttftMs/decodeMs/genTotalMs; usage += stepId; tool-result += durationMs (exec); done += durationMs (turn wall-clock) + usage (turn total). RunTurnInput += now?. [email protected] (re-export bump). - kernel-runtime: when now injected, measures + emits the above (reuses the ttft/decode first-token detection); omits timing gracefully without a clock. - session-orchestrator: adds now? to deps, threads into RunTurnInput; extension activate injects () => Date.now(). - transport/cli/host-bin: untouched (verbatim pass-through; additive fields). FE handoff: frontend-metrics-handoff.md. typecheck clean; 520 vitest + 89 bun; biome 0/0. Replay/persistence = deferred Pass 2 (documented in tasks.md).
Diffstat (limited to 'tasks.md')
-rw-r--r--tasks.md47
1 files changed, 47 insertions, 0 deletions
diff --git a/tasks.md b/tasks.md
index 89662b9..46849cb 100644
--- a/tasks.md
+++ b/tasks.md
@@ -580,6 +580,53 @@ measured** (provider-agnostic, self-consistent); **first token = first text OR r
spans with valid `durationMs` (ttft 1090ms, decode 1673ms) + `firstToken:true`. GLOSSARY: TTFT,
decode time. NOT on the wire (clients don't receive it) — a future wire+FE step if desired.
+### Expose backend metrics to clients (timing/tokens) — IN PROGRESS
+User ask: surface the backend's authoritative metrics (tokens, TTFT/decode, TPS, tool-exec +
+turn durations) to clients (CLI/web FE). Decisions (user, §5.2):
+- **Delivery = inline in the existing chat stream**, as distinct `AgentEvent` types/fields —
+ push-everything, FE routes by `type` and ignores what it doesn't render (NOT a separate live
+ endpoint, NOT a negotiation handshake — events are tiny/low-frequency, not per-token).
+- **Carrier = a new `step-complete` event** (per-step end, ordering-safe timing) + additive fields.
+- **Scope = LIVE now**; persisted-for-replay is a documented fast-follow (below).
+
+#### Pass 1 — LIVE stream metrics [~] IN PROGRESS
+Metric set (all wire fields additive/optional): per-step tokens (`usage` += `stepId`), per-step
+`ttftMs`/`decodeMs`/`genTotalMs` (new `step-complete` event), tool-exec `durationMs` (`tool-result`),
+turn `durationMs` + aggregate `usage` (`done`). TPS derived FE-side (`outputTokens/decodeMs`);
+context-size proxy = `usage.inputTokens` (already present).
+- [x] **Contracts (orchestrator):** `@dispatch/wire` — new `TurnStepCompleteEvent` (+ union);
+ `stepId?` on `TurnUsageEvent`; `durationMs?` on `TurnToolResultEvent`; `durationMs?`+`usage?` on
+ `TurnDoneEvent`. Kernel re-export shims (events.ts/index.ts) updated. `RunTurnInput.now?: () =>
+ number` (additive optional clock — runtime has no clock today; needed to put real numbers on the
+ wire). **Whole-graph typecheck clean** → new variant breaks NO consumer (no exhaustive switches;
+ cli/transport unaffected). GLOSSARY: TTFT/decode already added.
+- [ ] **Build wave (2 disjoint owner-agents, parallel, mimo-v2.5-pro):**
+ - **kernel-runtime:** when `now` provided, measure + emit timing — `step-complete` per step
+ (ttft/decode/genTotal, reusing the just-built first-token detection), `stepId` on `usage`,
+ `durationMs` on `tool-result`, `durationMs`+`usage` on `done`. Keep the trace spans (may unify
+ span timing with the numeric measurement). Omit timing gracefully when `now` absent. +tests.
+ - **session-orchestrator:** add `now?` to `SessionOrchestratorDeps`, thread into the
+ `RunTurnInput` it builds; provide `() => Date.now()` from the extension's `activate` (shell
+ edge). +test asserting forwarding.
+ - **NOT touched (typecheck-confirmed):** transport-http/ws (verbatim pass-through), cli (optional
+ fields, non-exhaustive switch), host-bin (orchestrator self-injects the clock in activate).
+- [ ] **Post-wave (orchestrator):** full typecheck/test/biome; bump `wire`+`transport-contract`
+ minor (`0.2.0→0.3.0`); regen FE `.dispatch/{wire,transport-contract}.reference.md`; courier reply.
+ Live: confirm `step-complete`/durations/`usage.stepId` on the real stream.
+
+#### Pass 2 — DEFERRED: persisted metrics for REPLAY
+Today usage/timing are NOT persisted (conversation-store stores chunks only), so reopening a past
+conversation (`GET /conversations/:id`) shows messages but no metrics. To show historical
+tokens/timing:
+- Persist a per-turn (and/or per-step) metrics record in `conversation-store` (new storage shape —
+ metrics are not chunks; e.g. a `conv:<id>:turn:<turnId>:metrics` key or a sibling namespace).
+- Expose on the read path: a `metrics` field on `ConversationHistoryResponse`, or a sibling
+ `GET /conversations/:id/metrics` (read endpoint = the right home for historical metrics, vs the
+ live stream). New contract + transport-http route + conversation-store schema + FE.
+- The trace-store already holds rich per-span timing durably; Pass 2 could alternatively expose a
+ read slice of THAT rather than duplicating into conversation-store (decide at Pass-2 design time).
+Sequenced after Pass 1 lands (and can ride alongside the FE-Slice work that consumes Pass 1).
+
### 3. dedup / storage growth (after frontend)
The deferred trace-body de-duplication + rotation/compression (D5 volume-control +
`prefix.fingerprint` + §6 retention strategy) — already designed in