diff --git a/docs/edge-telemetry.md b/docs/edge-telemetry.md new file mode 100644 index 0000000..73f8ecd --- /dev/null +++ b/docs/edge-telemetry.md @@ -0,0 +1,231 @@ +# Edge Telemetry — What token-meter Collects + +All data is read from local JSONL logs on the developer's machine. No API keys. +No data leaves the machine. + +## Log Sources + +| Provider | Path | Client | +|---|---|---| +| Claude Code CLI | `~/.claude/projects/*/*.jsonl` | `claude_code` | +| Claude Desktop (project) | `~/Library/Application Support/Claude/claude-code-sessions/` | `claude_desktop` | +| Claude Desktop (agent) | `~/Library/Application Support/Claude/local-agent-mode-sessions/` | `claude_desktop` | + +--- + +## Fields Captured + +### Tokens (`meter.py:684–706`) + +All four token types are read from the `usage` block of each assistant message: + +```python +def cost_of(u, model, provider="claude"): + p, _ = price_for(model, provider) + return { + "input": u.get("input_tokens", 0) * p["input"] / 1e6, + "cache_write": u.get("cache_creation_input_tokens", 0) * p["cache_write"] / 1e6, + "cache_read": u.get("cache_read_input_tokens", 0) * p["cache_read"] / 1e6, + "output": u.get("output_tokens", 0) * p["output"] / 1e6, + } + +def usage_tokens(u): + return (u.get("input_tokens", 0) + u.get("cache_creation_input_tokens", 0) + + u.get("cache_read_input_tokens", 0) + u.get("output_tokens", 0)) + +def usage_io_tokens(u): + return ( + int(u.get("input_tokens", 0) or 0) + + int(u.get("cache_creation_input_tokens", 0) or 0) + + int(u.get("cache_read_input_tokens", 0) or 0), + int(u.get("output_tokens", 0) or 0), + ) +``` + +| Field | Meaning | +|---|---| +| `input_tokens` | Fresh prompt tokens (not cached) | +| `output_tokens` | Generated tokens (includes thinking tokens — not broken out) | +| `cache_creation_input_tokens` | Tokens written to the prompt cache | +| `cache_read_input_tokens` | Tokens served from cache (cheaper rate) | + +Cost is computed locally by multiplying each token type by the hardcoded rate in `CLAUDE_PRICE` + +--- + +### Model and Provider (`meter.py:526–570`) + +Set at session discovery time from the file path and optional Desktop sidecar metadata: + +```python +# Claude sessions +source = { + "provider": "claude", + "client": desktop.get("client") or "claude_code", # or "claude_desktop" + "model": desktop.get("model"), + ... +} +``` + +--- + +### Session Timing and Output Throughput (`meter.py:154–212`) + +Claude reads `system` records with `subtype: "turn_duration"`: + +```python +if obj.get("type") != "system" or obj.get("subtype") != "turn_duration": + ... +duration_ms = obj.get("durationMs") +if ts and duration_ms > 0: + intervals.append((ts - duration_ms / 1000.0, ts)) +``` + +Output tokens ÷ generation seconds = observed tokens/sec, shown in the Stats tab. + +--- + +### Tool Calls and Tool Result Volume (`meter.py:1331–1350`) + +Tool call count is inferred from `tool_use` blocks in assistant messages. Tool result +volume is measured in characters from `tool_result` blocks returned by the user turn: + +```python +def claude_tool_results(objs): + chars_by_id = defaultdict(int) + for obj in objs: + if obj.get("type") != "user": + continue + for block in content: + if block.get("type") == "tool_result": + tid = block.get("tool_use_id") + chars_by_id[tid] += observable_output_chars(block.get("content", "")) +``` + +Errors per tool call are also tracked via `is_error` and heuristic content inspection. + +--- + +### Context Window Pressure (`meter.py:1617, 1659, 1783, 1796`) + +`model_context_window` is read from the log and used to compute fill percentage per execution: + +```python +context_window = None + +# Both providers: +context_pct = (in_tok / context_window) if context_window else None +``` + +Warning thresholds: 65% (soft), 70% (watch), 85% (intervene) — defined as +`MENUBAR_CONTEXT_SOFT_PCT`, `MENUBAR_CONTEXT_WATCH_PCT`, `MENUBAR_CONTEXT_INTERVENE_PCT`. + +--- + +### Working Directory / Project (`meter.py:535–536, 563`) + +```python +# Claude: read from JSONL trace records +trace_cwd = claude_trace_cwd(path) +project = desktop.get("project") or home_shorten(trace_cwd) or decode_claude_project(project_raw) +``` + +--- + +## ROI Signals (Currently Computed) + +| Signal | Formula | meter.py location | +|---|---|---| +| Cache efficiency | `cache_read_tokens / total_input_tokens` | `meter.py:1538` | +| Low-yield detection | `output_tokens / input_tokens < 0.005` | `meter.py:1021` | +| Low-yield warning | fires if context >25%, input >60K tokens, or 2+ consecutive low-yield turns | `meter.py:1031` | +| Cost spike | single execution ≥ $0.50 or ≥ 55% of session spend | `meter.py:2223` | +| Output throughput | `output_tokens / generation_seconds` | `meter.py:898` | +| Context pressure | `input_tokens / model_context_window` — warns at 65% / 70% / 85% | `meter.py:1796` | +| Tool result volume | characters returned per tool call; oversized flagged at >8K tokens | `meter.py:1345` | + +--- + +## Claude Gaps — Fields in Raw JSONL Not Yet Extracted + +Implemented in `meter.py` as `state.edge_telemetry` (Claude sessions only). + +### Session-level fields on every `user` record + +```json +{ + "type": "user", + "gitBranch": "main", + "entrypoint": "cli", + "version": "2.1.196" +} +``` + +| Field | Notes | +|---|---| +| `gitBranch` | Git branch active when the turn was sent | +| `entrypoint` | `cli` or `vscode` — how Claude Code was launched | +| `version` | Claude Code CLI version (e.g. `2.1.196`) | + +### Extra fields in the `usage` block + +```json +{ + "service_tier": "standard", + "speed": "standard", + "cache_creation": { + "ephemeral_5m_input_tokens": 15107, + "ephemeral_1h_input_tokens": 0 + }, + "server_tool_use": { + "web_search_requests": 0, + "web_fetch_requests": 0 + } +} +``` + +| Field | Notes | +|---|---| +| `service_tier` | `"standard"` — billing tier for the request | +| `speed` | `"standard"` or `"fast"` — model speed setting | +| `cache_creation.ephemeral_5m_input_tokens` | Short-lived cache tier (5 min) | +| `cache_creation.ephemeral_1h_input_tokens` | Longer-lived cache tier (1 hr) | +| `server_tool_use.web_search_requests` | Web search tool calls this request | +| `server_tool_use.web_fetch_requests` | Web fetch tool calls this request | + +### `stop_reason` on assistant messages + +Present as `message.stop_reason` but not aggregated into a completion rate. + +| Value | Meaning | +|---|---| +| `end_turn` | Model finished naturally — task complete signal | +| `tool_use` | Turn ended to invoke a tool — session still in progress | +| `max_tokens` | Hit token limit — potential truncation | + +--- + +## Claude Gaps — Derived Metrics Not Yet Computed + +Implemented under `state.edge_telemetry.derived` and `state.edge_telemetry.stop_reasons`. + +| Metric | How to derive | Where to look in JSONL | +|---|---|---| +| Task completion rate | `stop_reason == "end_turn"` ÷ total sessions | `message.stop_reason` on assistant records | +| Tokens per tool call | total session tokens ÷ tool call count | `usage` blocks + `tool_use` block count | +| Tokens per turn | total session tokens ÷ user message count | `usage` blocks + `type == "user"` count | +| Files edited per session | unique `path` args in `Edit`/`Write` tool calls | `tool_use` blocks where `name` is `Edit` or `Write` | +| Repeat queries | hash normalized user message text, compare against `~/.claude/history.jsonl` | `type == "user"` message content | + +--- + +## What Is Not Available at the Edge + +| Data point | Notes | +|---|---| +| Lines of code / commits / PRs | Only in Anthropic's cloud Analytics API | +| Accepted vs rejected edits | Not in JSONL — exists only in Cursor/Copilot UI layer | +| Cross-user / org-wide view | Each machine only sees its own logs | +| Claude.ai web usage | No local logs written | +| Claude iOS / mobile usage | No local logs written | +| Thinking tokens (separate count) | Folded into `output_tokens`; not broken out | diff --git a/meter.py b/meter.py index 0fa2ba2..f2942d4 100644 --- a/meter.py +++ b/meter.py @@ -37,6 +37,8 @@ CLAUDE_DESKTOP_SESSIONS = os.path.join(CLAUDE_DESKTOP_DATA_ROOTS[0], "claude-code-sessions") CLAUDE_SETTINGS = os.path.expanduser("~/.claude/settings.json") CLAUDE_ROOT_CONFIG = os.path.expanduser("~/.claude.json") +CLAUDE_HISTORY = os.path.expanduser("~/.claude/history.jsonl") +_HISTORY_QUERY_CACHE = {"mtime": None, "counts": {}} CODEX_SESSIONS = os.path.expanduser("~/.codex/sessions") CODEX_INDEX = os.path.expanduser("~/.codex/session_index.jsonl") CODEX_CONFIG = os.path.expanduser("~/.codex/config.toml") @@ -1428,6 +1430,173 @@ def claude_user_events(objs): return sorted(events, key=lambda e: e["ts"]) +def normalize_query_text(text): + return " ".join(str(text or "").lower().split()) + + +def query_fingerprint(text): + normalized = normalize_query_text(text) + if not normalized: + return "" + return hashlib.sha256(normalized.encode("utf-8", "replace")).hexdigest()[:16] + + +def claude_history_query_counts(): + """Return normalized query fingerprint counts from Claude history.jsonl.""" + global _HISTORY_QUERY_CACHE + try: + mtime = os.path.getmtime(CLAUDE_HISTORY) + except OSError: + return {} + if _HISTORY_QUERY_CACHE["mtime"] == mtime: + return _HISTORY_QUERY_CACHE["counts"] + counts = defaultdict(int) + for obj in load(CLAUDE_HISTORY): + fp = query_fingerprint(obj.get("display") or "") + if fp: + counts[fp] += 1 + counts = dict(counts) + _HISTORY_QUERY_CACHE = {"mtime": mtime, "counts": counts} + return counts + + +def claude_session_context(objs): + """Latest session-level metadata from Claude user records.""" + ctx = {"git_branch": None, "entrypoint": None, "version": None, "branches_seen": []} + branches = set() + for obj in objs: + if obj.get("type") != "user": + continue + branch = obj.get("gitBranch") + if branch: + branches.add(str(branch)) + ctx["git_branch"] = str(branch) + entrypoint = obj.get("entrypoint") + if entrypoint: + ctx["entrypoint"] = str(entrypoint) + version = obj.get("version") + if version: + ctx["version"] = str(version) + ctx["branches_seen"] = sorted(branches) + return ctx + + +def claude_usage_extras(msgs): + """Aggregate Claude usage fields beyond the four billing buckets.""" + extras = { + "service_tiers": {}, + "speeds": {}, + "cache_ephemeral_5m": 0, + "cache_ephemeral_1h": 0, + "web_search_requests": 0, + "web_fetch_requests": 0, + } + for rec in msgs: + usage = rec.get("usage") or {} + tier = usage.get("service_tier") + if tier: + key = str(tier) + extras["service_tiers"][key] = extras["service_tiers"].get(key, 0) + 1 + speed = usage.get("speed") + if speed: + key = str(speed) + extras["speeds"][key] = extras["speeds"].get(key, 0) + 1 + cache_creation = usage.get("cache_creation") or {} + extras["cache_ephemeral_5m"] += int(cache_creation.get("ephemeral_5m_input_tokens") or 0) + extras["cache_ephemeral_1h"] += int(cache_creation.get("ephemeral_1h_input_tokens") or 0) + server_tool_use = usage.get("server_tool_use") or {} + extras["web_search_requests"] += int(server_tool_use.get("web_search_requests") or 0) + extras["web_fetch_requests"] += int(server_tool_use.get("web_fetch_requests") or 0) + return extras + + +def claude_stop_reason_summary(msgs): + counts = defaultdict(int) + for rec in msgs: + reason = rec.get("stop_reason") + if reason: + counts[str(reason)] += 1 + total = sum(counts.values()) + end_turn = counts.get("end_turn", 0) + return { + "counts": dict(counts), + "total": total, + "end_turn": end_turn, + "tool_use": counts.get("tool_use", 0), + "max_tokens": counts.get("max_tokens", 0), + "completion_rate": (end_turn / total) if total else 0.0, + } + + +def claude_edited_files(msgs): + paths = set() + for rec in msgs: + for block in rec.get("content") or []: + if not isinstance(block, dict) or block.get("type") != "tool_use": + continue + name = str(block.get("name") or "") + if name not in ("Edit", "Write"): + continue + inp = block.get("input") + if not isinstance(inp, dict): + continue + path = inp.get("file_path") or inp.get("path") + if path: + paths.add(str(path)) + return sorted(paths) + + +def claude_human_turn_count(objs): + return sum(1 for obj in objs if _claude_human_text(obj)) + + +def claude_repeat_queries(objs, history_counts=None): + """Count human user turns that repeat a prior query in history or this session.""" + history_counts = history_counts if history_counts is not None else claude_history_query_counts() + seen_in_session = set() + human_turns = 0 + repeat_queries = 0 + for obj in objs: + text = _claude_human_text(obj) + if not text or not str(text).strip(): + continue + human_turns += 1 + fp = query_fingerprint(text) + if not fp: + continue + if fp in seen_in_session or history_counts.get(fp, 0) > 1: + repeat_queries += 1 + seen_in_session.add(fp) + return { + "human_turns": human_turns, + "repeat_queries": repeat_queries, + "repeat_rate": (repeat_queries / human_turns) if human_turns else 0.0, + } + + +def claude_edge_telemetry(objs, msgs, total_tokens, tool_calls): + """Extract documented Claude JSONL gaps and derived session metrics.""" + human_turns = claude_human_turn_count(objs) + stop_reasons = claude_stop_reason_summary(msgs) + repeats = claude_repeat_queries(objs) + edited_files = claude_edited_files(msgs) + return { + "session": claude_session_context(objs), + "usage": claude_usage_extras(msgs), + "stop_reasons": stop_reasons, + "derived": { + "completion_rate": stop_reasons["completion_rate"], + "tokens_per_tool_call": (total_tokens / tool_calls) if tool_calls else None, + "tokens_per_turn": (total_tokens / human_turns) if human_turns else None, + "files_edited": len(edited_files), + "human_turns": human_turns, + "repeat_queries": repeats["repeat_queries"], + "repeat_rate": repeats["repeat_rate"], + }, + "edited_files": edited_files, + } + + def tool_summary(executions): by_name = {} by_namespace = {} @@ -1809,6 +1978,7 @@ def recompute_claude(source): "reasoning": out_tok if has_think else 0, "user_message": user_input, "user_input": user_input, + "stop_reason": rec.get("stop_reason"), }) executions.append({ "id": rec["id"], @@ -1832,6 +2002,7 @@ def recompute_claude(source): "summary": f"Turn {idx}: {out_tok:,} out / {in_tok:,} in", "user_message": user_input, "user_input": user_input, + "stop_reason": rec.get("stop_reason"), }) if biggest is None or tc > biggest["cost"]: biggest = {"cost": tc, "idx": idx} @@ -1862,6 +2033,9 @@ def recompute_claude(source): analyses, insights, first_ts, last_ts, idle, biggest, side_turns, approx_cost, primary_model, "exact Claude API-rate estimate", execution_timing("claude", objs)) state["throughput"] = performance_summary(claude_performance_samples(objs), tot["output"]) + state["edge_telemetry"] = claude_edge_telemetry( + objs, msgs, total_tokens, tool_data["total_calls"], + ) return state diff --git a/page.html b/page.html index 74cfe52..e7c21ea 100644 --- a/page.html +++ b/page.html @@ -66,11 +66,12 @@ .section-h{display:flex;align-items:center;gap:10px;margin:24px 0 10px} .section-h h2{font-size:14px;margin:0;font-weight:780} .section-h .hint{font-size:12px;color:var(--faint)} -.sessionhead{display:flex;align-items:center;gap:9px;margin-bottom:10px;min-width:0} +.sessionhead{display:flex;align-items:center;gap:9px;margin-bottom:10px;min-width:0;flex-wrap:wrap} .sessionStart{display:grid;grid-template-columns:max-content minmax(0,1fr);gap:9px;align-items:baseline;margin-bottom:12px;padding:9px 11px;border:1px solid var(--line);border-left:3px solid var(--accent);border-radius:8px;background:rgba(0,188,235,.055);min-width:0}.sessionStart[hidden]{display:none}.sessionStart span{color:var(--faint);font-size:10.5px;font-weight:800;text-transform:uppercase;white-space:nowrap}.sessionStart strong{font-size:12.5px;font-weight:680;line-height:1.4;overflow-wrap:anywhere} .repo,.sessid{border:1px solid var(--line);background:rgba(17,24,32,.76);border-radius:8px;padding:6px 10px;font-size:12px;min-width:0;box-shadow:var(--inset)} .repo{font-weight:700;overflow:hidden;text-overflow:ellipsis;white-space:nowrap} .sessid{color:var(--dim);max-width:360px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap} +.sessid.wrap,.ov .sm.wrap{max-width:none;overflow:visible;text-overflow:unset;white-space:normal;line-height:1.4;overflow-wrap:anywhere} .unpin{display:none;border:1px solid var(--line2);background:rgba(17,24,32,.9);border-radius:8px;color:var(--dim);font:inherit;font-size:12px;padding:6px 11px;cursor:pointer;box-shadow:var(--inset)} .unpin.on{display:inline-block}.unpin:hover{color:var(--fg);border-color:var(--accent)} .sessionDelete{border:1px solid rgba(255,111,111,.36);background:rgba(255,111,111,.07);border-radius:8px;color:#ffaaaa;font:inherit;font-size:12px;font-weight:720;padding:6px 10px;cursor:pointer;box-shadow:var(--inset);white-space:nowrap}.sessionDelete:hover:not(:disabled){border-color:rgba(255,111,111,.68);background:rgba(255,111,111,.13);color:#ffd0d0}.sessionDelete:disabled{opacity:.42;cursor:not-allowed}.sessionDelete[hidden]{display:none}.smetrics .sessionDelete{font-size:10.5px;padding:4px 7px;margin-top:5px} @@ -197,6 +198,7 @@ source -- + @@ -260,7 +262,7 @@