diff options
| author | Adam Malczewski <[email protected]> | 2026-06-10 10:43:40 +0900 |
|---|---|---|
| committer | Adam Malczewski <[email protected]> | 2026-06-10 10:43:40 +0900 |
| commit | 3a0cdd2c8453f059a746465e3aa6d9b5caa3b399 (patch) | |
| tree | b76133d2a593ccd510673a745e5912599f850696 /tasks.md | |
| parent | 80e14ab59732aabbf06035d13138500f133e921d (diff) | |
| download | dispatch-3a0cdd2c8453f059a746465e3aa6d9b5caa3b399.tar.gz dispatch-3a0cdd2c8453f059a746465e3aa6d9b5caa3b399.zip | |
trace-store: content-addressed body dedup + retention/prune
Wave 1 of the dedup/storage-growth milestone (notes §12).
- bodies table is now content-addressed (SHA-256 hash key); identical verbatim
bodies (cache-warming resends, any repeat) collapse to one stored row,
referenced by hash from records. Transparent to insert/read callers.
- at-rest gzip compression for bodies >1 KiB (node:zlib), decompressed on read.
- prune(policy): age-based delete + drop-oldest byte-cap eviction + orphan-body
GC. Exports RetentionPolicy/PruneSummary/DEFAULT_RETENTION (7d / 256 MiB).
typecheck EXIT 0; biome clean; vitest 576; bun 89->100, 0 fail.
Diffstat (limited to 'tasks.md')
| -rw-r--r-- | tasks.md | 28 |
1 files changed, 23 insertions, 5 deletions
@@ -5,7 +5,7 @@ > Keep this lean and current; do not let it re-accrete a step-by-step changelog. ## Status (current) -`tsc -b` EXIT 0 · biome clean · **546 vitest + 89 bun = 635 tests**. +`tsc -b` EXIT 0 · biome clean · **576 vitest + 89 bun = 665 tests**. Built and verified live (full-fidelity: every feature is a manifest-loaded extension through the host): @@ -68,11 +68,29 @@ server/collector procs poison the next run's counts. - [x] **FE courier handoff** written: `frontend-metrics-pass2-handoff.md` (in this repo; user couriers to `../dispatch-web`; ORCHESTRATOR §7). +## dedup / storage growth (current milestone — building) +Design DECIDED + recorded: `notes/observability-design.md` §12. User-gated calls: +extend existing pipeline (no new ext); scope = **de-dup + retention/rotation** +(D9 roll-ups deferred); dedup = **content-addressed bodies** (body-hash, NOT +fingerprint-gated). Glossary terms approved (add on land): deduplication / +content-addressed body, prefix fingerprint, warm vs real, retention / rotation. +- [x] **Wave 1 — `trace-store`** (done): content-addressed `bodies` table + (SHA-256), at-rest gzip (>1 KiB), `prune(policy)` (age + drop-oldest size cap + + orphan GC) / `RetentionPolicy` / `PruneSummary` / `DEFAULT_RETENTION` (7d/256MiB); + read paths transparent. bun 89→100. +- [ ] **Wave 2 — `observability-collector`:** call `store.prune()` on a cadence; + body inserts flow through the content-addressed path. +- [ ] On land: add the 4 glossary terms; (optional) host-bin env-override for + retention policy. + ## Open items -- **dedup / storage growth (deferred):** trace-body de-dup (D5 volume control + - `prefix.fingerprint`) + rotation/compression/retention - (`notes/observability-design.md` §6, D9). `cacheReadTokens` is the cheap dedup - signal; thin/fat split already built. +- **`prefix.fingerprint` / `warm|real` cache-bust attributes (deferred):** decoupled + from dedup by the content-addressed decision; also gated on cache-warming being + built (not yet) so `warm|real` can't be honestly stamped. Later cache-bust-debug + milestone (`notes/observability-design.md` §3.1, §12). +- **D9 analytics roll-ups (deferred):** rollup table shape + `GROUP BY` indexes + + retention asymmetry + periodic rollup job (`notes/observability-design.md` §2 D9, + §12). The scheduler mechanism (`host.scheduler.register`) already exists. - **D8 `prompt.assembly` segments:** deferred-by-design (await the context-filter chain). |
