diff --git a/README.md b/README.md index ad7a5a3..ec75c86 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # tokitoki-cli -TokiToki is a small cross-platform uploader for local AI coding usage. Each +Tokitoki is a small cross-platform uploader for local AI coding usage. Each run reads the configured local agent usage folders, then uploads the discovered -events to the local TokiToki server. +events to the local Tokitoki server. The CLI can run once or install itself as an OS service. @@ -78,7 +78,7 @@ service status Print service status. ## Stable project names -TokiToki normally uses the project name reported by an IDE or local AI agent. +Tokitoki normally uses the project name reported by an IDE or local AI agent. To give a checkout a stable name across editors, machines, and differently named local folders, create `.tokitoki` in the project root: @@ -90,7 +90,7 @@ release/2026 The first line overrides the project name. The optional second line overrides the branch. An empty file uses the containing folder's name and preserves any branch reported by the editor. The nearest project file found by walking up -from the active file wins; for out-of-tree agent files, TokiToki also searches +from the active file wins; for out-of-tree agent files, Tokitoki also searches the event's reported project path. Use `{project}` in the first line to include the nearest Git, Mercurial, or @@ -107,8 +107,7 @@ folder containing `.tokitoki`. Only regular files are read: the shared data directory `~/.tokitoki` shares the name but is a directory, so it is never mistaken for a project file. -Only `.tokitoki` is read. legacy `.legacy-project` and `.toolconfig` -files are not treated as project names by TokiToki. +Only `.tokitoki` is read; other tools' project files are ignored. Project-file resolution is applied centrally before events enter the local queue, so it affects IDE heartbeats and AI-agent usage scans consistently. It @@ -154,7 +153,7 @@ tokitoki service install --system ## The shared CLI -Every TokiToki front-end — the macOS and Windows apps and every editor +Every Tokitoki front-end — the macOS and Windows apps and every editor plugin — invokes one shared copy of this CLI. Its location is a contract, not a suggestion; a plugin that resolves a different path forks the fleet and stops receiving updates: diff --git a/TODO.md b/TODO.md index 922e02e..0884d07 100644 --- a/TODO.md +++ b/TODO.md @@ -1,4 +1,4 @@ -# TokiToki TODO +# Tokitoki TODO ## Known Issues (prioritized, reviewed 2026-07-17) @@ -19,7 +19,7 @@ P0 — storage consolidation (done 2026-07-17): P0 — dead scaffolding after the daemon→CLI refactor: - [x] Remove the heartbeat chain. It POSTed to `/heartbeats`, which the - TokiToki server does not implement (only `/api/usage-events/batch` exists). + Tokitoki server does not implement (only `/api/usage-events/batch` exists). Removed: `Heartbeat` type, `RecordHeartbeat`/`Sync`/`postHeartbeats`, `queue.jsonl` + store queue methods, and the `heartbeat`/`push-heartbeats` subcommands. (A new `heartbeat` subcommand exists today, but it feeds the @@ -34,7 +34,7 @@ P1 — correctness/robustness: - [x] Upload watermark: each event now has local upload state, so `tokitoki` uploads only pending/failed events and marks accepted/duplicate server responses as uploaded. -- [ ] Claude streaming token upsert (see legacy note below) — affects count +- [ ] Claude streaming token upsert (see notes below) — affects count accuracy. - [ ] Replace hardcoded `InstallationID: "local-go-agent"` with a per-machine id persisted under `~/.tokitoki/`, so one user's multiple devices stay distinct. @@ -42,7 +42,7 @@ P1 — correctness/robustness: P2 — missing basics: - [x] `version` command / `--version`, with the version injected via ldflags. -- [ ] Scan lock + large-transcript protection (see legacy notes below). +- [ ] Scan lock + large-transcript protection (see notes below). P3 — cross-platform packaging: @@ -53,12 +53,12 @@ P3 — cross-platform packaging: ## Scanner Ideas To Revisit -- Add a scan cursor similar to legacy `ai_logs_last_parsed_at` so unchanged provider files are skipped. (The old `source_files` table was dead code and has been removed; a fresh design would add per-file rows to `usage.db`.) +- Add a per-file scan cursor so unchanged provider files are skipped. (The old `source_files` table was dead code and has been removed; a fresh design would add per-file rows to `usage.db`.) - Add a short global scan lock so startup scan and manual `/usage/scan` cannot parse the same files concurrently. - Add large transcript protection: bounded scanner buffers, max line size handling, and optional tail scanning for very large session files. - Fix Claude streaming-update semantics: use a logical event key based on `provider + session_id + request_id + message_id`, then upsert latest token values instead of inserting a new event when token counts grow. -- Review Codex token handling against legacy total-token/delta approach and decide whether TokiToki should store raw token-count events or normalized deltas. -- ~~Add upload queue behavior inspired by legacy offline sync~~ — done 2026-07-17: batch limits, stop-on-first-failure, exponential backoff, permanent rejection, and pruning all live in `usagedb` + `usageupload.SyncPending`. +- Review Codex token handling and decide whether Tokitoki should store raw token-count events or normalized deltas. +- ~~Add offline upload queue behavior~~ — done 2026-07-17: batch limits, stop-on-first-failure, exponential backoff, permanent rejection, and pruning all live in `usagedb` + `usageupload.SyncPending`. - Add summary-level upload state so server sync sends daily/project/model summaries instead of every local `usage_event`. - Add source status/debug endpoints for scan counts, last scan time, last error, indexed event count, and pending upload count. diff --git a/cmd/tokitoki/main.go b/cmd/tokitoki/main.go index 2273480..1a90436 100644 --- a/cmd/tokitoki/main.go +++ b/cmd/tokitoki/main.go @@ -1,5 +1,5 @@ // Command tokitoki scans local Claude Code/Codex usage files and uploads the -// resulting events to the local TokiToki server. +// resulting events to the local Tokitoki server. package main import ( @@ -547,7 +547,7 @@ func copyProviderDirs(providerDirs map[agentlib.Provider][]string) map[agentlib. } func usage() { - fmt.Fprint(os.Stderr, `tokitoki — upload local AI usage to the TokiToki server + fmt.Fprint(os.Stderr, `tokitoki — upload local AI usage to the Tokitoki server Usage: tokitoki [--provider-dir PROVIDER=DIR ...] [--check-update] @@ -561,7 +561,7 @@ Usage: tokitoki service [options] Each invocation scans the provider roots you pass and uploads their usage -events to the TokiToki server (TOKITOKI_BASE_URL, default +events to the Tokitoki server (TOKITOKI_BASE_URL, default https://tokitoki.dev). By default, tokitoki scans the built-in roots for claude, codex, copilot, gemini, kimi, qwen, openclaw, pi, amp, droid, kilo, hermes, codebuff, opencode, and goose. Pass one or more diff --git a/cmd/tokitoki/service_linux.go b/cmd/tokitoki/service_linux.go index 49690d5..513abb4 100644 --- a/cmd/tokitoki/service_linux.go +++ b/cmd/tokitoki/service_linux.go @@ -127,7 +127,7 @@ func systemdUnitTexts(flags workerFlags, system bool) (string, string, error) { var service strings.Builder service.WriteString("[Unit]\n") - service.WriteString("Description=Sync local AI usage to TokiToki\n") + service.WriteString("Description=Sync local AI usage to Tokitoki\n") service.WriteString("Wants=network-online.target\n") service.WriteString("After=network-online.target\n\n") service.WriteString("[Service]\n") @@ -145,7 +145,7 @@ func systemdUnitTexts(flags workerFlags, system bool) (string, string, error) { seconds = 1 } timer := fmt.Sprintf(`[Unit] -Description=Run the TokiToki usage sync on an interval +Description=Run the Tokitoki usage sync on an interval [Timer] OnBootSec=2min diff --git a/cmd/tokitoki/service_other.go b/cmd/tokitoki/service_other.go index 8d47133..b87a2e5 100644 --- a/cmd/tokitoki/service_other.go +++ b/cmd/tokitoki/service_other.go @@ -54,8 +54,8 @@ func newService(flags workerFlags, userService bool) (daemonservice.Service, err } config := &daemonservice.Config{ Name: "tokitoki", - DisplayName: "TokiToki", - Description: "Sync local AI usage to TokiToki.", + DisplayName: "Tokitoki", + Description: "Sync local AI usage to Tokitoki.", Executable: executable, Arguments: serviceArguments(flags), Option: daemonservice.KeyValue{ diff --git a/internal/agent/agent.go b/internal/agent/agent.go index c3005a1..18af448 100644 --- a/internal/agent/agent.go +++ b/internal/agent/agent.go @@ -6,10 +6,6 @@ import ( type Settings struct { APIKey string `json:"api_key"` - // InstallationID is this install's stable random identity. The server - // keys device rows on it, so it must be unique per machine and constant - // across runs — the store generates it once and persists it. - InstallationID string `json:"installation_id"` } type Store interface { diff --git a/internal/agentusage/amp.go b/internal/agentusage/amp.go index 3024782..2377afd 100644 --- a/internal/agentusage/amp.go +++ b/internal/agentusage/amp.go @@ -10,7 +10,7 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadAmpEntries(paths []string) ([]usage.Entry, error) { +func loadAmpEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, path := range paths { if info, err := os.Stat(path); err == nil && !info.IsDir() { @@ -25,7 +25,7 @@ func loadAmpEntries(paths []string) ([]usage.Entry, error) { } } sort.Strings(files) - files = uniqueStrings(files) + files = filterFiles(uniqueStrings(files), filter) entries := make([]usage.Entry, 0) for _, file := range files { diff --git a/internal/agentusage/codebuff.go b/internal/agentusage/codebuff.go index a047d92..6bac979 100644 --- a/internal/agentusage/codebuff.go +++ b/internal/agentusage/codebuff.go @@ -22,13 +22,13 @@ type codebuffUsage struct { extraTotalTokens uint64 } -func loadCodebuffEntries(paths []string) ([]usage.Entry, error) { +func loadCodebuffEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, root := range paths { files = append(files, collectCodebuffChatFiles(root)...) } sort.Strings(files) - files = uniqueStrings(files) + files = filterFiles(uniqueStrings(files), filter) entriesByID := make(map[string]usage.Entry) for _, file := range files { diff --git a/internal/agentusage/copilot.go b/internal/agentusage/copilot.go index 170169e..e2aaefe 100644 --- a/internal/agentusage/copilot.go +++ b/internal/agentusage/copilot.go @@ -10,13 +10,13 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadCopilotEntries(paths []string) ([]usage.Entry, error) { +func loadCopilotEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, root := range paths { files = append(files, collectExt(root, ".jsonl")...) } sort.Strings(files) - files = uniqueStrings(files) + files = filterFiles(uniqueStrings(files), filter) entries := make([]usage.Entry, 0) for _, file := range files { diff --git a/internal/agentusage/droid.go b/internal/agentusage/droid.go index d6932b3..fe6a4a7 100644 --- a/internal/agentusage/droid.go +++ b/internal/agentusage/droid.go @@ -10,13 +10,13 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadDroidEntries(paths []string) ([]usage.Entry, error) { +func loadDroidEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, root := range paths { files = append(files, collectFiles(root, isDroidSettingsFile)...) } sort.Strings(files) - files = uniqueStrings(files) + files = filterFiles(uniqueStrings(files), filter) entries := make([]usage.Entry, 0) for _, file := range files { diff --git a/internal/agentusage/gemini.go b/internal/agentusage/gemini.go index f29bf1b..8fa999a 100644 --- a/internal/agentusage/gemini.go +++ b/internal/agentusage/gemini.go @@ -9,14 +9,14 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadGeminiEntries(paths []string) ([]usage.Entry, error) { +func loadGeminiEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, root := range paths { files = append(files, collectExt(root, ".json")...) files = append(files, collectExt(root, ".jsonl")...) } sort.Strings(files) - files = uniqueStrings(files) + files = filterFiles(uniqueStrings(files), filter) entries := make([]usage.Entry, 0) for _, file := range files { diff --git a/internal/agentusage/helpers.go b/internal/agentusage/helpers.go index 0dd38e5..9a26e53 100644 --- a/internal/agentusage/helpers.go +++ b/internal/agentusage/helpers.go @@ -55,6 +55,20 @@ func collectFiles(root string, match func(string) bool) []string { return files } +// filterFiles drops files the filter rejects. A nil filter keeps everything. +func filterFiles(files []string, filter usage.FileFilter) []string { + if filter == nil { + return files + } + kept := files[:0] + for _, file := range files { + if filter(file) { + kept = append(kept, file) + } + } + return kept +} + func collectExt(root, ext string) []string { ext = strings.ToLower(ext) return collectFiles(root, func(path string) bool { diff --git a/internal/agentusage/kimi.go b/internal/agentusage/kimi.go index a3b21c6..b24ca7f 100644 --- a/internal/agentusage/kimi.go +++ b/internal/agentusage/kimi.go @@ -8,13 +8,13 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadKimiEntries(paths []string) ([]usage.Entry, error) { +func loadKimiEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, root := range paths { files = append(files, collectFiles(filepath.Join(root, "sessions"), isKimiWireFile)...) } sort.Strings(files) - files = uniqueStrings(files) + files = filterFiles(uniqueStrings(files), filter) entries := make([]usage.Entry, 0) seen := make(map[string]bool) diff --git a/internal/agentusage/openclaw.go b/internal/agentusage/openclaw.go index e3f8098..30bf4d3 100644 --- a/internal/agentusage/openclaw.go +++ b/internal/agentusage/openclaw.go @@ -8,12 +8,13 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadOpenClawEntries(paths []string) ([]usage.Entry, error) { +func loadOpenClawEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, path := range paths { files = append(files, collectFiles(path, isOpenClawSessionFile)...) } sort.Strings(files) + files = filterFiles(files, filter) entries := make([]usage.Entry, 0) for _, file := range files { diff --git a/internal/agentusage/opencode.go b/internal/agentusage/opencode.go index cbf4629..a0c2b4b 100644 --- a/internal/agentusage/opencode.go +++ b/internal/agentusage/opencode.go @@ -12,10 +12,10 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadOpenCodeEntries(paths []string) ([]usage.Entry, error) { +func loadOpenCodeEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { entriesByID := make(map[string]usage.Entry) for _, root := range paths { - rootEntries, err := loadOpenCodeRoot(root) + rootEntries, err := loadOpenCodeRoot(root, filter) if err != nil { return nil, err } @@ -36,7 +36,7 @@ func loadOpenCodeEntries(paths []string) ([]usage.Entry, error) { return entries, nil } -func loadOpenCodeRoot(root string) ([]usage.Entry, error) { +func loadOpenCodeRoot(root string, filter usage.FileFilter) ([]usage.Entry, error) { info, err := os.Stat(root) if err != nil { return nil, nil @@ -73,6 +73,7 @@ func loadOpenCodeRoot(root string) ([]usage.Entry, error) { files := collectExt(filepath.Join(root, "storage", "message"), ".json") sort.Strings(files) + files = filterFiles(files, filter) for _, file := range files { stem := strings.TrimSuffix(filepath.Base(file), filepath.Ext(file)) if seenIDs[stableOpenCodeMessageID(stem)] { diff --git a/internal/agentusage/pi.go b/internal/agentusage/pi.go index 9253475..304eeac 100644 --- a/internal/agentusage/pi.go +++ b/internal/agentusage/pi.go @@ -8,12 +8,13 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadPiEntries(paths []string) ([]usage.Entry, error) { +func loadPiEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, path := range paths { files = append(files, collectExt(path, ".jsonl")...) } sort.Strings(files) + files = filterFiles(files, filter) entries := make([]usage.Entry, 0) for _, file := range files { diff --git a/internal/agentusage/providers.go b/internal/agentusage/providers.go index 0646d41..4c2c52f 100644 --- a/internal/agentusage/providers.go +++ b/internal/agentusage/providers.go @@ -5,8 +5,14 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usageprovider" ) +// providerBase carries the scan configuration shared by every agent +// provider. filter skips source files whose events are already ingested; +// the SQLite-backed providers (Kilo, Hermes, Goose) deliberately do not +// implement WithFileFilter because WAL keeps the main database file's stat +// unchanged while data grows in the -wal journal. type providerBase struct { - paths []string + paths []string + filter usage.FileFilter } func newProviderBase(paths []string) providerBase { @@ -76,9 +82,16 @@ func (CopilotProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the GitHub Copilot CLI provider id. func (CopilotProvider) Provider() usage.Provider { return usage.ProviderCopilot } +// WithFileFilter returns a GitHub Copilot CLI provider that skips source +// files the filter rejects. +func (p CopilotProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized GitHub Copilot CLI usage entries. func (p CopilotProvider) Entries() ([]usage.Entry, error) { - return loadCopilotEntries(p.paths) + return loadCopilotEntries(p.paths, p.filter) } // WithPaths returns a Gemini CLI provider configured with data roots. @@ -89,9 +102,16 @@ func (GeminiProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the Gemini CLI provider id. func (GeminiProvider) Provider() usage.Provider { return usage.ProviderGemini } +// WithFileFilter returns a Gemini CLI provider that skips source files the +// filter rejects. +func (p GeminiProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized Gemini CLI usage entries. func (p GeminiProvider) Entries() ([]usage.Entry, error) { - return loadGeminiEntries(p.paths) + return loadGeminiEntries(p.paths, p.filter) } // WithPaths returns a Kimi provider configured with data roots. @@ -102,9 +122,16 @@ func (KimiProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the Kimi provider id. func (KimiProvider) Provider() usage.Provider { return usage.ProviderKimi } +// WithFileFilter returns a Kimi provider that skips source files the filter +// rejects. +func (p KimiProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized Kimi usage entries. func (p KimiProvider) Entries() ([]usage.Entry, error) { - return loadKimiEntries(p.paths) + return loadKimiEntries(p.paths, p.filter) } // WithPaths returns a Qwen provider configured with data roots. @@ -115,9 +142,16 @@ func (QwenProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the Qwen provider id. func (QwenProvider) Provider() usage.Provider { return usage.ProviderQwen } +// WithFileFilter returns a Qwen provider that skips source files the filter +// rejects. +func (p QwenProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized Qwen usage entries. func (p QwenProvider) Entries() ([]usage.Entry, error) { - return loadQwenEntries(p.paths) + return loadQwenEntries(p.paths, p.filter) } // WithPaths returns an OpenClaw provider configured with data roots. @@ -128,9 +162,16 @@ func (OpenClawProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the OpenClaw provider id. func (OpenClawProvider) Provider() usage.Provider { return usage.ProviderOpenClaw } +// WithFileFilter returns an OpenClaw provider that skips source files the +// filter rejects. +func (p OpenClawProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized OpenClaw usage entries. func (p OpenClawProvider) Entries() ([]usage.Entry, error) { - return loadOpenClawEntries(p.paths) + return loadOpenClawEntries(p.paths, p.filter) } // WithPaths returns a pi-agent provider configured with data roots. @@ -141,9 +182,16 @@ func (PiProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the pi-agent provider id. func (PiProvider) Provider() usage.Provider { return usage.ProviderPi } +// WithFileFilter returns a pi-agent provider that skips source files the +// filter rejects. +func (p PiProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized pi-agent usage entries. func (p PiProvider) Entries() ([]usage.Entry, error) { - return loadPiEntries(p.paths) + return loadPiEntries(p.paths, p.filter) } // WithPaths returns an Amp provider configured with data roots. @@ -154,9 +202,16 @@ func (AmpProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the Amp provider id. func (AmpProvider) Provider() usage.Provider { return usage.ProviderAmp } +// WithFileFilter returns an Amp provider that skips source files the filter +// rejects. +func (p AmpProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized Amp usage entries. func (p AmpProvider) Entries() ([]usage.Entry, error) { - return loadAmpEntries(p.paths) + return loadAmpEntries(p.paths, p.filter) } // WithPaths returns a Droid provider configured with data roots. @@ -167,9 +222,16 @@ func (DroidProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the Droid provider id. func (DroidProvider) Provider() usage.Provider { return usage.ProviderDroid } +// WithFileFilter returns a Droid provider that skips source files the filter +// rejects. +func (p DroidProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized Droid usage entries. func (p DroidProvider) Entries() ([]usage.Entry, error) { - return loadDroidEntries(p.paths) + return loadDroidEntries(p.paths, p.filter) } // WithPaths returns a Kilo provider configured with data roots. @@ -206,9 +268,16 @@ func (CodebuffProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the Codebuff provider id. func (CodebuffProvider) Provider() usage.Provider { return usage.ProviderCodebuff } +// WithFileFilter returns a Codebuff provider that skips source files the +// filter rejects. +func (p CodebuffProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized Codebuff usage entries. func (p CodebuffProvider) Entries() ([]usage.Entry, error) { - return loadCodebuffEntries(p.paths) + return loadCodebuffEntries(p.paths, p.filter) } // WithPaths returns an OpenCode provider configured with data roots. @@ -219,9 +288,17 @@ func (OpenCodeProvider) WithPaths(paths []string) usageprovider.Provider { // Provider returns the OpenCode provider id. func (OpenCodeProvider) Provider() usage.Provider { return usage.ProviderOpenCode } +// WithFileFilter returns an OpenCode provider that skips source files the +// filter rejects. The filter applies to message files only; the OpenCode +// database goes through the SQLite path and is always scanned. +func (p OpenCodeProvider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p +} + // Entries loads normalized OpenCode usage entries. func (p OpenCodeProvider) Entries() ([]usage.Entry, error) { - return loadOpenCodeEntries(p.paths) + return loadOpenCodeEntries(p.paths, p.filter) } // WithPaths returns a Goose provider configured with data roots. diff --git a/internal/agentusage/providers_test.go b/internal/agentusage/providers_test.go index def897e..c3f0a04 100644 --- a/internal/agentusage/providers_test.go +++ b/internal/agentusage/providers_test.go @@ -9,6 +9,29 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) +func TestProviderFileFilterSkipsRejectedFiles(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "copilot.jsonl") + writeFile(t, path, `{"type":"span","traceId":"trace-1","spanId":"span-1","name":"chat claude-sonnet-4","endTime":[1775934264,967317833],"attributes":{"gen_ai.operation.name":"chat","gen_ai.response.model":"claude-sonnet-4","gen_ai.conversation.id":"conv-1","gen_ai.usage.input_tokens":19452,"gen_ai.usage.output_tokens":281}}}`+"\n") + + rejected := make([]string, 0) + provider := CopilotProvider{}.WithPaths([]string{dir}).(CopilotProvider). + WithFileFilter(func(candidate string) bool { + rejected = append(rejected, candidate) + return false + }) + entries, err := provider.Entries() + if err != nil { + t.Fatal(err) + } + if len(entries) != 0 { + t.Fatalf("len(entries) = %d, want 0 when the filter rejects every file", len(entries)) + } + if len(rejected) != 1 || rejected[0] != path { + t.Fatalf("filter saw %#v, want the session file", rejected) + } +} + func TestProvidersLoadEntries(t *testing.T) { tests := []struct { name string diff --git a/internal/agentusage/qwen.go b/internal/agentusage/qwen.go index 3d43d2d..36c2a40 100644 --- a/internal/agentusage/qwen.go +++ b/internal/agentusage/qwen.go @@ -8,7 +8,7 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/usage" ) -func loadQwenEntries(paths []string) ([]usage.Entry, error) { +func loadQwenEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) { files := make([]string, 0) for _, root := range paths { files = append(files, collectFiles(filepath.Join(root, "projects"), isQwenChatFile)...) @@ -17,7 +17,7 @@ func loadQwenEntries(paths []string) ([]usage.Entry, error) { } } sort.Strings(files) - files = uniqueStrings(files) + files = filterFiles(uniqueStrings(files), filter) entries := make([]usage.Entry, 0) for _, file := range files { diff --git a/internal/claudeusage/loader.go b/internal/claudeusage/loader.go index 4e10dad..00eae27 100644 --- a/internal/claudeusage/loader.go +++ b/internal/claudeusage/loader.go @@ -172,10 +172,13 @@ func SummarizeDailyProjects(entries []LoadedEntry) []DailyProjectSummary { return summaries } -func LoadEntriesFromPaths(paths []string, projectFilter string) ([]LoadedEntry, error) { +func LoadEntriesFromPaths(paths []string, projectFilter string, fileFilter usage.FileFilter) ([]LoadedEntry, error) { files := UsageFiles(paths, projectFilter) entries := make([]LoadedEntry, 0) for _, file := range files { + if fileFilter != nil && !fileFilter(file) { + continue + } fileEntries, err := ReadUsageFile(file) if err != nil { return nil, err diff --git a/internal/claudeusage/loader_test.go b/internal/claudeusage/loader_test.go index 4b8dad6..fa1387b 100644 --- a/internal/claudeusage/loader_test.go +++ b/internal/claudeusage/loader_test.go @@ -225,7 +225,7 @@ func TestLoadEntriesDeduplicatesByMessageAndRequest(t *testing.T) { {"timestamp":"2026-05-21T01:02:04Z","requestId":"req-1","message":{"id":"msg-1","model":"claude","usage":{"input_tokens":10,"output_tokens":1}}} `) - entries, err := LoadEntriesFromPaths([]string{dir}, "") + entries, err := LoadEntriesFromPaths([]string{dir}, "", nil) if err != nil { t.Fatal(err) } diff --git a/internal/claudeusage/provider.go b/internal/claudeusage/provider.go index 9c6710e..27fa81c 100644 --- a/internal/claudeusage/provider.go +++ b/internal/claudeusage/provider.go @@ -7,14 +7,23 @@ import ( // Provider loads Claude usage entries. type Provider struct { - paths []string + paths []string + filter usage.FileFilter } var _ usageprovider.Provider = Provider{} // WithPaths returns a Claude provider configured with data roots. -func (Provider) WithPaths(paths []string) usageprovider.Provider { - return Provider{paths: append([]string{}, paths...)} +func (p Provider) WithPaths(paths []string) usageprovider.Provider { + p.paths = append([]string{}, paths...) + return p +} + +// WithFileFilter returns a Claude provider that skips session files the +// filter rejects. +func (p Provider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p } // Provider returns the Claude provider id. @@ -24,7 +33,7 @@ func (Provider) Provider() usage.Provider { // Entries loads normalized Claude usage entries. func (p Provider) Entries() ([]usage.Entry, error) { - entries, err := LoadEntriesFromPaths(p.paths, "") + entries, err := LoadEntriesFromPaths(p.paths, "", p.filter) if err != nil { return nil, err } diff --git a/internal/cli/cli.go b/internal/cli/cli.go index ab679e2..4f35843 100644 --- a/internal/cli/cli.go +++ b/internal/cli/cli.go @@ -1,4 +1,4 @@ -// Package cli implements TokiToki's one operation: scan local AI usage files +// Package cli implements Tokitoki's one operation: scan local AI usage files // and upload them to the local server. package cli diff --git a/internal/codexusage/loader.go b/internal/codexusage/loader.go index 5687180..9b90bd7 100644 --- a/internal/codexusage/loader.go +++ b/internal/codexusage/loader.go @@ -52,10 +52,13 @@ type tokenUsagePayload struct { TotalTokens uint64 `json:"total_tokens"` } -func LoadEntriesFromPaths(paths []string, projectFilter string) ([]usage.Entry, error) { +func LoadEntriesFromPaths(paths []string, projectFilter string, fileFilter usage.FileFilter) ([]usage.Entry, error) { files := UsageFiles(paths) entries := make([]usage.Entry, 0) for _, file := range files { + if fileFilter != nil && !fileFilter(file) { + continue + } fileEntries, err := ReadUsageFile(file) if err != nil { return nil, err diff --git a/internal/codexusage/loader_test.go b/internal/codexusage/loader_test.go index 139adf6..f51e3c6 100644 --- a/internal/codexusage/loader_test.go +++ b/internal/codexusage/loader_test.go @@ -108,7 +108,7 @@ func TestLoadEntriesFiltersByProjectOrProjectPath(t *testing.T) { {"timestamp":"2026-06-03T01:02:05Z","type":"event_msg","payload":{"type":"token_count","info":{"last_token_usage":{"input_tokens":1,"output_tokens":2,"total_tokens":3}}}} `) - entries, err := LoadEntriesFromPaths([]string{dir}, "tokitoki") + entries, err := LoadEntriesFromPaths([]string{dir}, "tokitoki", nil) if err != nil { t.Fatal(err) } @@ -116,7 +116,7 @@ func TestLoadEntriesFiltersByProjectOrProjectPath(t *testing.T) { t.Fatalf("len(entries) = %d, want 1", len(entries)) } - entries, err = LoadEntriesFromPaths([]string{dir}, "/Users/me/workspace/tokitoki") + entries, err = LoadEntriesFromPaths([]string{dir}, "/Users/me/workspace/tokitoki", nil) if err != nil { t.Fatal(err) } @@ -124,7 +124,7 @@ func TestLoadEntriesFiltersByProjectOrProjectPath(t *testing.T) { t.Fatalf("len(entries by path) = %d, want 1", len(entries)) } - entries, err = LoadEntriesFromPaths([]string{dir}, "other") + entries, err = LoadEntriesFromPaths([]string{dir}, "other", nil) if err != nil { t.Fatal(err) } diff --git a/internal/codexusage/provider.go b/internal/codexusage/provider.go index 65f70ca..fad493b 100644 --- a/internal/codexusage/provider.go +++ b/internal/codexusage/provider.go @@ -7,14 +7,23 @@ import ( // Provider loads Codex usage entries. type Provider struct { - paths []string + paths []string + filter usage.FileFilter } var _ usageprovider.Provider = Provider{} // WithPaths returns a Codex provider configured with data roots. -func (Provider) WithPaths(paths []string) usageprovider.Provider { - return Provider{paths: append([]string{}, paths...)} +func (p Provider) WithPaths(paths []string) usageprovider.Provider { + p.paths = append([]string{}, paths...) + return p +} + +// WithFileFilter returns a Codex provider that skips session files the +// filter rejects. +func (p Provider) WithFileFilter(filter usage.FileFilter) usageprovider.Provider { + p.filter = filter + return p } // Provider returns the Codex provider id. @@ -24,5 +33,5 @@ func (Provider) Provider() usage.Provider { // Entries loads normalized Codex usage entries. func (p Provider) Entries() ([]usage.Entry, error) { - return LoadEntriesFromPaths(p.paths, "") + return LoadEntriesFromPaths(p.paths, "", p.filter) } diff --git a/internal/config/config.go b/internal/config/config.go index e6e7f81..007ab3f 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -2,7 +2,7 @@ package config const ( // DataDirName is the hidden directory created under the user's home - // directory for shared TokiToki agent state. The same path is used on + // directory for shared Tokitoki agent state. The same path is used on // macOS, Windows, and Linux so every native front-end resolves it the // same way: filepath.Join(os.UserHomeDir(), config.DataDirName). DataDirName = ".tokitoki" diff --git a/internal/langdetect/langdetect_test.go b/internal/langdetect/langdetect_test.go index 091c5fa..4202bcd 100644 --- a/internal/langdetect/langdetect_test.go +++ b/internal/langdetect/langdetect_test.go @@ -2,7 +2,7 @@ package langdetect import "testing" -func TestFromPathUseslegacyStyleFilenameAndExtensionRules(t *testing.T) { +func TestFromPathUsesFilenameAndExtensionRules(t *testing.T) { tests := map[string]string{ "/repo/go.mod": "Go", "/repo/app/page.tsx": "TypeScript", diff --git a/internal/projectfile/projectfile.go b/internal/projectfile/projectfile.go index c44c9fa..27dcca1 100644 --- a/internal/projectfile/projectfile.go +++ b/internal/projectfile/projectfile.go @@ -13,7 +13,7 @@ import ( ) const ( - // Name is TokiToki's canonical per-project configuration file, placed in + // Name is Tokitoki's canonical per-project configuration file, placed in // the project root. The shared data directory ~/.tokitoki shares the name // but is a directory; the lookup only accepts regular files, so the two // never collide. diff --git a/internal/projectfile/projectfile_test.go b/internal/projectfile/projectfile_test.go index 27b486d..b7374e2 100644 --- a/internal/projectfile/projectfile_test.go +++ b/internal/projectfile/projectfile_test.go @@ -155,7 +155,7 @@ func TestResolvePlaceholderUsesProjectPathVCSForOutOfTreeEntity(t *testing.T) { } } -func TestResolveIgnoreslegacyProjectFiles(t *testing.T) { +func TestResolveIgnoresOtherToolsProjectFiles(t *testing.T) { root := t.TempDir() outer := filepath.Join(root, "outer") inner := filepath.Join(outer, "inner") @@ -198,7 +198,7 @@ func TestResolveNoFilePreservesExistingDetection(t *testing.T) { } } -func TestResolveDoesNotTreatDotlegacyConfigAsProjectIdentity(t *testing.T) { +func TestResolveDoesNotTreatOtherDotfilesAsProjectIdentity(t *testing.T) { projectDir, entity := projectTree(t) mustWriteFile(t, filepath.Join(projectDir, ".toolconfig"), "[settings]\nproject=not-an-identity\n") @@ -207,7 +207,7 @@ func TestResolveDoesNotTreatDotlegacyConfigAsProjectIdentity(t *testing.T) { t.Fatal(err) } if found || result != (Result{}) { - t.Fatalf("Resolve() = (%+v, %t), want .toolconfig ignored", result, found) + t.Fatalf("Resolve() = (%+v, %t), want other dotfiles ignored", result, found) } } diff --git a/internal/selfupdate/selfupdate.go b/internal/selfupdate/selfupdate.go index a144f0f..8dfc79c 100644 --- a/internal/selfupdate/selfupdate.go +++ b/internal/selfupdate/selfupdate.go @@ -31,7 +31,7 @@ import ( "github.com/tokitoki-dev/tokitoki-cli/internal/store" ) -// UpdateChannel is the release channel the CLI reads on the TokiToki server. +// UpdateChannel is the release channel the CLI reads on the Tokitoki server. const UpdateChannel = "cli" const ( @@ -53,7 +53,7 @@ type Result struct { // "dev" above all — identifies a local build, which must never update itself. var semverRE = regexp.MustCompile(`^v?[0-9]+\.[0-9]+\.[0-9]+`) -// Upgrade brings the running executable up to date against the TokiToki +// Upgrade brings the running executable up to date against the Tokitoki // server. baseURL is the server root (usageupload.BaseURL()); current is the // running version. // diff --git a/internal/selfupdate/selfupdate_test.go b/internal/selfupdate/selfupdate_test.go index 18f21ed..62b85ef 100644 --- a/internal/selfupdate/selfupdate_test.go +++ b/internal/selfupdate/selfupdate_test.go @@ -138,7 +138,7 @@ func TestUpgradeRejectsVersionMismatch(t *testing.T) { } func TestUpgradeRefusesAppBundle(t *testing.T) { - dir := filepath.Join(t.TempDir(), "TokiToki.app", "Contents", "Resources") + dir := filepath.Join(t.TempDir(), "Tokitoki.app", "Contents", "Resources") if err := os.MkdirAll(dir, 0o755); err != nil { t.Fatal(err) } diff --git a/internal/store/lock.go b/internal/store/lock.go index c4d3790..a6e71a6 100644 --- a/internal/store/lock.go +++ b/internal/store/lock.go @@ -18,7 +18,7 @@ const UploadLockFile = "upload.lock" // ErrLockBusy reports that another process held the lock for the whole // timeout. Callers that treat "someone else is already doing this work" as // success test for it with errors.Is. -var ErrLockBusy = errors.New("another TokiToki command is still running") +var ErrLockBusy = errors.New("another Tokitoki command is still running") type DataLock struct { file *os.File diff --git a/internal/store/store.go b/internal/store/store.go index 8ea52fa..554cff2 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -1,10 +1,7 @@ package store import ( - "crypto/rand" - "encoding/hex" "errors" - "fmt" "os" "path/filepath" "strings" @@ -15,9 +12,8 @@ import ( ) const ( - UsageDBFile = "usage.db" - apiKeyFile = "api_key" - installIDFile = "installation_id" + UsageDBFile = "usage.db" + apiKeyFile = "api_key" directoryMod = 0o700 apiKeyFileMod = 0o600 ) @@ -53,56 +49,22 @@ func Open(dir string) (*FileStore, error) { return &FileStore{dir: dir}, nil } -// LoadSettings reads the API key from the api_key file and this install's -// stable identity, generating the latter on first use. +// LoadSettings reads the API key from the api_key file. func (s *FileStore) LoadSettings() (agent.Settings, error) { s.mu.Lock() defer s.mu.Unlock() - installID, err := s.installationIDLocked() - if err != nil { - return agent.Settings{}, err - } - data, err := os.ReadFile(filepath.Join(s.dir, apiKeyFile)) if errors.Is(err, os.ErrNotExist) { if err := s.ensureAPIKeyFileLocked(); err != nil { return agent.Settings{}, err } - return agent.Settings{InstallationID: installID}, nil + return agent.Settings{}, nil } if err != nil { return agent.Settings{}, err } - return agent.Settings{ - APIKey: strings.TrimSpace(string(data)), - InstallationID: installID, - }, nil -} - -// installationIDLocked returns the install's stable random identity, minting -// and persisting one the first time it is asked for. The server keys device -// rows on this value, so it must never change once written. -func (s *FileStore) installationIDLocked() (string, error) { - path := filepath.Join(s.dir, installIDFile) - data, err := os.ReadFile(path) - if err == nil { - if id := strings.TrimSpace(string(data)); id != "" { - return id, nil - } - } else if !errors.Is(err, os.ErrNotExist) { - return "", err - } - - raw := make([]byte, 16) - if _, err := rand.Read(raw); err != nil { - return "", fmt.Errorf("generate installation id: %w", err) - } - id := hex.EncodeToString(raw) - if err := s.writeFileLocked(path, id); err != nil { - return "", fmt.Errorf("persist installation id: %w", err) - } - return id, nil + return agent.Settings{APIKey: strings.TrimSpace(string(data))}, nil } func (s *FileStore) EnsureAPIKeyFile() error { diff --git a/internal/usage/timezone.go b/internal/usage/timezone.go new file mode 100644 index 0000000..4053dce --- /dev/null +++ b/internal/usage/timezone.go @@ -0,0 +1,63 @@ +package usage + +import ( + "os" + "strings" + "time" +) + +// MachineTimezone reports the IANA zone this machine is in, or "" when it +// genuinely cannot be determined. +// +// Why this is not one line of standard library: Go's time.Local reports the +// literal string "Local" on every platform, and the only name it will hand out +// is an abbreviation ("JST"), which is ambiguous — IST is India, Ireland and +// Israel; CST is China, US Central and Cuba. An abbreviation cannot be expanded +// back into a zone, so it is useless for analysis. +// +// The name is therefore looked for in three places, most authoritative first: +// +// 1. $TZ, because that is what Go itself obeys — when it is set it *is* this +// process's zone, whatever the OS thinks. +// 2. time.Local's own name, on the off chance a platform populates it. +// 3. The OS's own record of the choice, which is where the answer actually +// lives: the /etc/localtime symlink on Unix, the registry on Windows. +// See timezone_unix.go and timezone_windows.go. +// +// Callers must handle "". A stripped container with no tzdata and no TZ set has +// no zone to report, and inventing one would be worse than admitting it. +func MachineTimezone() string { + if name := os.Getenv("TZ"); name != "" { + // A leading colon is legal in TZ (":Asia/Tokyo") and names a file path + // by convention; strip it before validating. + name = strings.TrimPrefix(name, ":") + if isLoadableZone(name) { + return name + } + } + // "Local" and "UTC" are Go's placeholders rather than answers. UTC is + // rejected deliberately: a machine reporting it is far more often one with + // no zone configured than one genuinely in Greenwich, and a confidently + // wrong name is worse than none. + if name := time.Local.String(); name != "" && name != "Local" && name != "UTC" { + if isLoadableZone(name) { + return name + } + } + return systemZoneName() +} + +// isLoadableZone is the gate every candidate passes before being reported. +// +// It keeps a name this machine's tzdata cannot resolve from being sent as fact: +// the server would only discard it, and Postgres raises outright on an unknown +// name in AT TIME ZONE. The "path traversal" check matters because $TZ is +// attacker-controllable in some deployments and Go resolves it against the +// zoneinfo directory. +func isLoadableZone(name string) bool { + if name == "" || strings.Contains(name, "..") { + return false + } + _, err := time.LoadLocation(name) + return err == nil +} diff --git a/internal/usage/timezone_test.go b/internal/usage/timezone_test.go new file mode 100644 index 0000000..5caf62f --- /dev/null +++ b/internal/usage/timezone_test.go @@ -0,0 +1,70 @@ +package usage + +import ( + "testing" + "time" +) + +func TestMachineTimezoneUsesTZWhenSet(t *testing.T) { + t.Setenv("TZ", "Asia/Tokyo") + if got := MachineTimezone(); got != "Asia/Tokyo" { + t.Fatalf("MachineTimezone() = %q, want Asia/Tokyo", got) + } +} + +func TestMachineTimezoneAcceptsLeadingColon(t *testing.T) { + // ":Asia/Tokyo" is legal TZ syntax; the colon names a file path by + // convention and is not part of the zone name. + t.Setenv("TZ", ":Asia/Tokyo") + if got := MachineTimezone(); got != "Asia/Tokyo" { + t.Fatalf("MachineTimezone() = %q, want Asia/Tokyo", got) + } +} + +func TestMachineTimezoneRejectsJunkTZ(t *testing.T) { + // A bogus TZ must not be reported as fact. Falling through to the system + // source is fine; echoing "Not/AZone" is not — the server would discard it + // and Postgres raises on it. + t.Setenv("TZ", "Not/AZone") + if got := MachineTimezone(); got == "Not/AZone" { + t.Fatal("MachineTimezone() reported an unloadable zone name") + } +} + +func TestMachineTimezoneRejectsPathTraversal(t *testing.T) { + t.Setenv("TZ", "../../etc/passwd") + if got := MachineTimezone(); got == "../../etc/passwd" { + t.Fatal("MachineTimezone() echoed a traversal path") + } +} + +func TestMachineTimezoneReturnsLoadableNameOrEmpty(t *testing.T) { + // The contract callers rely on: whatever comes back either loads as a real + // zone or is empty. Never an abbreviation, never a guess. + got := MachineTimezone() + if got == "" { + t.Skip("no zone resolvable on this machine") + } + if _, err := time.LoadLocation(got); err != nil { + t.Fatalf("MachineTimezone() = %q, which does not load: %v", got, err) + } + if got == "Local" { + t.Fatal("MachineTimezone() returned Go's placeholder rather than a name") + } +} + +func TestIsLoadableZone(t *testing.T) { + for _, name := range []string{"UTC", "Asia/Tokyo", "America/New_York"} { + if !isLoadableZone(name) { + t.Errorf("isLoadableZone(%q) = false, want true", name) + } + } + // "JST" is the case that matters: Go will hand out this abbreviation, and + // it must never be reported, because IST/CST/BST style abbreviations map to + // several different real zones. + for _, name := range []string{"", "Not/AZone", "../etc/passwd", "JST"} { + if isLoadableZone(name) { + t.Errorf("isLoadableZone(%q) = true, want false", name) + } + } +} diff --git a/internal/usage/timezone_unix.go b/internal/usage/timezone_unix.go new file mode 100644 index 0000000..ae1515f --- /dev/null +++ b/internal/usage/timezone_unix.go @@ -0,0 +1,61 @@ +//go:build !windows + +package usage + +import ( + "os" + "path/filepath" + "strings" +) + +// zoneinfoMarker is the one path segment every tzdata layout shares: the IANA +// name is whatever follows it. +// +// Matching on this rather than a fixed prefix is what makes the lookup work on +// macOS, where /etc/localtime resolves through a versioned directory — +// /private/var/db/timezone/tz/2026b.1.0/zoneinfo/Asia/Tokyo — that no +// hard-coded prefix would match. Linux resolves to /usr/share/zoneinfo/... and +// falls out of the same rule. +const zoneinfoMarker = "/zoneinfo/" + +// systemZoneName reads the IANA name out of /etc/localtime, which is where Unix +// records the machine's zone. +// +// Returns "" when /etc/localtime is a copied file rather than a symlink (some +// minimal images do this, and a copy carries no name), or when the extracted +// name is not one tzdata knows. +func systemZoneName() string { + resolved, err := filepath.EvalSymlinks("/etc/localtime") + if err != nil { + return "" + } + if name := zoneNameFromPath(resolved); name != "" { + return name + } + // Some distributions record the name in a plain text file instead. Debian + // has always had /etc/timezone; several others follow it. + for _, path := range []string{"/etc/timezone"} { + contents, err := os.ReadFile(path) + if err != nil { + continue + } + name := strings.TrimSpace(string(contents)) + if isLoadableZone(name) { + return name + } + } + return "" +} + +func zoneNameFromPath(resolved string) string { + resolved = filepath.ToSlash(resolved) + index := strings.LastIndex(resolved, zoneinfoMarker) + if index < 0 { + return "" + } + name := resolved[index+len(zoneinfoMarker):] + if !isLoadableZone(name) { + return "" + } + return name +} diff --git a/internal/usage/timezone_windows.go b/internal/usage/timezone_windows.go new file mode 100644 index 0000000..9926fc5 --- /dev/null +++ b/internal/usage/timezone_windows.go @@ -0,0 +1,57 @@ +//go:build windows + +package usage + +import ( + "golang.org/x/sys/windows/registry" +) + +// systemZoneName reads the machine's zone from the Windows registry and +// translates it into an IANA name. +// +// Windows keeps its own zone list rather than IANA's, so this is a two-step +// lookup: the registry holds a Windows key name ("Tokyo Standard Time"), and +// windows_zones.go maps that onto "Asia/Tokyo" using the Unicode CLDR table +// that Go itself ships. Without the translation the value would be useless to +// the server, which speaks IANA everywhere. +// +// TimeZoneKeyName is the value to read; StandardName is deliberately not used +// as a fallback because it is *localised* — on a Japanese-language Windows it +// holds "東京 (標準時)", which appears in no mapping table. TimeZoneKeyName is +// invariant across display languages, which is exactly what makes it usable. +// +// Returns "" if the key cannot be read (a locked-down profile, a stripped +// Windows image) or holds a name newer than this build's CLDR table. Reporting +// nothing is correct there — the caller treats a missing zone as unknown rather +// than guessing. +func systemZoneName() string { + key, err := registry.OpenKey( + registry.LOCAL_MACHINE, + `SYSTEM\CurrentControlSet\Control\TimeZoneInformation`, + registry.QUERY_VALUE, + ) + if err != nil { + return "" + } + defer key.Close() + + windowsName, _, err := key.GetStringValue("TimeZoneKeyName") + if err != nil { + return "" + } + // Windows has been observed to pad this value with trailing NULs. + windowsName = trimNulls(windowsName) + + iana, ok := windowsToIANA[windowsName] + if !ok || !isLoadableZone(iana) { + return "" + } + return iana +} + +func trimNulls(value string) string { + for len(value) > 0 && value[len(value)-1] == 0 { + value = value[:len(value)-1] + } + return value +} diff --git a/internal/usage/usage.go b/internal/usage/usage.go index fa4f2f8..c7c0fcf 100644 --- a/internal/usage/usage.go +++ b/internal/usage/usage.go @@ -30,6 +30,11 @@ const ( const UnknownLanguage = "Unknown" +// FileFilter reports whether a source file must be parsed. Returning false +// means the file's events are already ingested and parsing it again would be +// wasted work. A nil FileFilter parses everything. +type FileFilter func(path string) bool + type TokenUsage struct { InputTokens uint64 `json:"input_tokens"` OutputTokens uint64 `json:"output_tokens"` diff --git a/internal/usage/windows_zones.go b/internal/usage/windows_zones.go new file mode 100644 index 0000000..d2a38e3 --- /dev/null +++ b/internal/usage/windows_zones.go @@ -0,0 +1,156 @@ +// Code generated from Go's zoneinfo_abbrs_windows.go (itself generated from +// the Unicode CLDR windowsZones.xml). DO NOT EDIT. +// Regenerate with: python3 scripts/gen-windows-zones.py + +package usage + +// windowsToIANA maps a Windows time zone key name, as stored in the registry +// under HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation, to the +// IANA name the rest of the system speaks. +// +// Windows does not use IANA zones; it keeps its own list ("Tokyo Standard +// Time"), and the correspondence is maintained by the Unicode CLDR project. +// This table is the only way to obtain an IANA name on Windows: Go's +// time.Local reports the literal string "Local" on every platform, and the +// zone abbreviations it does expose ("JST") are ambiguous across regions. +var windowsToIANA = map[string]string{ + "AUS Central Standard Time": "Australia/Darwin", + "AUS Eastern Standard Time": "Australia/Sydney", + "Afghanistan Standard Time": "Asia/Kabul", + "Alaskan Standard Time": "America/Anchorage", + "Aleutian Standard Time": "America/Adak", + "Altai Standard Time": "Asia/Barnaul", + "Arab Standard Time": "Asia/Riyadh", + "Arabian Standard Time": "Asia/Dubai", + "Arabic Standard Time": "Asia/Baghdad", + "Argentina Standard Time": "America/Buenos_Aires", + "Astrakhan Standard Time": "Europe/Astrakhan", + "Atlantic Standard Time": "America/Halifax", + "Aus Central W. Standard Time": "Australia/Eucla", + "Azerbaijan Standard Time": "Asia/Baku", + "Azores Standard Time": "Atlantic/Azores", + "Bahia Standard Time": "America/Bahia", + "Bangladesh Standard Time": "Asia/Dhaka", + "Belarus Standard Time": "Europe/Minsk", + "Bougainville Standard Time": "Pacific/Bougainville", + "Canada Central Standard Time": "America/Regina", + "Cape Verde Standard Time": "Atlantic/Cape_Verde", + "Caucasus Standard Time": "Asia/Yerevan", + "Cen. Australia Standard Time": "Australia/Adelaide", + "Central America Standard Time": "America/Guatemala", + "Central Asia Standard Time": "Asia/Bishkek", + "Central Brazilian Standard Time": "America/Cuiaba", + "Central Europe Standard Time": "Europe/Budapest", + "Central European Standard Time": "Europe/Warsaw", + "Central Pacific Standard Time": "Pacific/Guadalcanal", + "Central Standard Time": "America/Chicago", + "Central Standard Time (Mexico)": "America/Mexico_City", + "Chatham Islands Standard Time": "Pacific/Chatham", + "China Standard Time": "Asia/Shanghai", + "Cuba Standard Time": "America/Havana", + "Dateline Standard Time": "Etc/GMT+12", + "E. Africa Standard Time": "Africa/Nairobi", + "E. Australia Standard Time": "Australia/Brisbane", + "E. Europe Standard Time": "Europe/Chisinau", + "E. South America Standard Time": "America/Sao_Paulo", + "Easter Island Standard Time": "Pacific/Easter", + "Eastern Standard Time": "America/New_York", + "Eastern Standard Time (Mexico)": "America/Cancun", + "Egypt Standard Time": "Africa/Cairo", + "Ekaterinburg Standard Time": "Asia/Yekaterinburg", + "FLE Standard Time": "Europe/Kiev", + "Fiji Standard Time": "Pacific/Fiji", + "GMT Standard Time": "Europe/London", + "GTB Standard Time": "Europe/Bucharest", + "Georgian Standard Time": "Asia/Tbilisi", + "Greenland Standard Time": "America/Godthab", + "Greenwich Standard Time": "Atlantic/Reykjavik", + "Haiti Standard Time": "America/Port-au-Prince", + "Hawaiian Standard Time": "Pacific/Honolulu", + "India Standard Time": "Asia/Calcutta", + "Iran Standard Time": "Asia/Tehran", + "Israel Standard Time": "Asia/Jerusalem", + "Jordan Standard Time": "Asia/Amman", + "Kaliningrad Standard Time": "Europe/Kaliningrad", + "Korea Standard Time": "Asia/Seoul", + "Libya Standard Time": "Africa/Tripoli", + "Line Islands Standard Time": "Pacific/Kiritimati", + "Lord Howe Standard Time": "Australia/Lord_Howe", + "Magadan Standard Time": "Asia/Magadan", + "Magallanes Standard Time": "America/Punta_Arenas", + "Marquesas Standard Time": "Pacific/Marquesas", + "Mauritius Standard Time": "Indian/Mauritius", + "Middle East Standard Time": "Asia/Beirut", + "Montevideo Standard Time": "America/Montevideo", + "Morocco Standard Time": "Africa/Casablanca", + "Mountain Standard Time": "America/Denver", + "Mountain Standard Time (Mexico)": "America/Mazatlan", + "Myanmar Standard Time": "Asia/Rangoon", + "N. Central Asia Standard Time": "Asia/Novosibirsk", + "Namibia Standard Time": "Africa/Windhoek", + "Nepal Standard Time": "Asia/Katmandu", + "New Zealand Standard Time": "Pacific/Auckland", + "Newfoundland Standard Time": "America/St_Johns", + "Norfolk Standard Time": "Pacific/Norfolk", + "North Asia East Standard Time": "Asia/Irkutsk", + "North Asia Standard Time": "Asia/Krasnoyarsk", + "North Korea Standard Time": "Asia/Pyongyang", + "Omsk Standard Time": "Asia/Omsk", + "Pacific SA Standard Time": "America/Santiago", + "Pacific Standard Time": "America/Los_Angeles", + "Pacific Standard Time (Mexico)": "America/Tijuana", + "Pakistan Standard Time": "Asia/Karachi", + "Paraguay Standard Time": "America/Asuncion", + "Qyzylorda Standard Time": "Asia/Qyzylorda", + "Romance Standard Time": "Europe/Paris", + "Russia Time Zone 10": "Asia/Srednekolymsk", + "Russia Time Zone 11": "Asia/Kamchatka", + "Russia Time Zone 3": "Europe/Samara", + "Russian Standard Time": "Europe/Moscow", + "SA Eastern Standard Time": "America/Cayenne", + "SA Pacific Standard Time": "America/Bogota", + "SA Western Standard Time": "America/La_Paz", + "SE Asia Standard Time": "Asia/Bangkok", + "Saint Pierre Standard Time": "America/Miquelon", + "Sakhalin Standard Time": "Asia/Sakhalin", + "Samoa Standard Time": "Pacific/Apia", + "Sao Tome Standard Time": "Africa/Sao_Tome", + "Saratov Standard Time": "Europe/Saratov", + "Singapore Standard Time": "Asia/Singapore", + "South Africa Standard Time": "Africa/Johannesburg", + "South Sudan Standard Time": "Africa/Juba", + "Sri Lanka Standard Time": "Asia/Colombo", + "Sudan Standard Time": "Africa/Khartoum", + "Syria Standard Time": "Asia/Damascus", + "Taipei Standard Time": "Asia/Taipei", + "Tasmania Standard Time": "Australia/Hobart", + "Tocantins Standard Time": "America/Araguaina", + "Tokyo Standard Time": "Asia/Tokyo", + "Tomsk Standard Time": "Asia/Tomsk", + "Tonga Standard Time": "Pacific/Tongatapu", + "Transbaikal Standard Time": "Asia/Chita", + "Turkey Standard Time": "Europe/Istanbul", + "Turks And Caicos Standard Time": "America/Grand_Turk", + "US Eastern Standard Time": "America/Indianapolis", + "US Mountain Standard Time": "America/Phoenix", + "UTC": "Etc/UTC", + "UTC+12": "Etc/GMT-12", + "UTC+13": "Etc/GMT-13", + "UTC-02": "Etc/GMT+2", + "UTC-08": "Etc/GMT+8", + "UTC-09": "Etc/GMT+9", + "UTC-11": "Etc/GMT+11", + "Ulaanbaatar Standard Time": "Asia/Ulaanbaatar", + "Venezuela Standard Time": "America/Caracas", + "Vladivostok Standard Time": "Asia/Vladivostok", + "Volgograd Standard Time": "Europe/Volgograd", + "W. Australia Standard Time": "Australia/Perth", + "W. Central Africa Standard Time": "Africa/Lagos", + "W. Europe Standard Time": "Europe/Berlin", + "W. Mongolia Standard Time": "Asia/Hovd", + "West Asia Standard Time": "Asia/Tashkent", + "West Bank Standard Time": "Asia/Hebron", + "West Pacific Standard Time": "Pacific/Port_Moresby", + "Yakutsk Standard Time": "Asia/Yakutsk", + "Yukon Standard Time": "America/Whitehorse", +} diff --git a/internal/usagedb/db.go b/internal/usagedb/db.go index d9ec5e9..94d6436 100644 --- a/internal/usagedb/db.go +++ b/internal/usagedb/db.go @@ -34,8 +34,21 @@ CREATE TABLE IF NOT EXISTS usage_events ( last_error TEXT NOT NULL DEFAULT '' ); CREATE INDEX IF NOT EXISTS idx_usage_events_queue ON usage_events(status, next_attempt_at); +CREATE TABLE IF NOT EXISTS scanned_files ( + path TEXT PRIMARY KEY, + size INTEGER NOT NULL, + mtime_ns INTEGER NOT NULL +); ` +// FileState is the stat snapshot of a source file at the time it was last +// successfully scanned. A file whose current stat matches its stored state +// holds no events the database has not already seen. +type FileState struct { + Size int64 + MtimeNS int64 +} + type DB struct { db *sql.DB } @@ -100,6 +113,53 @@ func (s *DB) InsertEvents(entries []usage.Entry) (int, error) { return inserted, tx.Commit() } +// ScannedFiles returns the stat snapshot of every file recorded as scanned. +func (s *DB) ScannedFiles() (map[string]FileState, error) { + rows, err := s.db.Query(`SELECT path, size, mtime_ns FROM scanned_files`) + if err != nil { + return nil, err + } + defer rows.Close() + + states := make(map[string]FileState) + for rows.Next() { + var path string + var state FileState + if err := rows.Scan(&path, &state.Size, &state.MtimeNS); err != nil { + return nil, err + } + states[path] = state + } + return states, rows.Err() +} + +// UpsertScannedFiles records the stat snapshots of files whose events have +// been ingested. Call it only after the corresponding InsertEvents succeeded; +// recording a file before its events are stored would skip them forever. +func (s *DB) UpsertScannedFiles(states map[string]FileState) error { + if len(states) == 0 { + return nil + } + tx, err := s.db.Begin() + if err != nil { + return err + } + defer tx.Rollback() + + stmt, err := tx.Prepare(`INSERT OR REPLACE INTO scanned_files (path, size, mtime_ns) VALUES (?, ?, ?)`) + if err != nil { + return err + } + defer stmt.Close() + + for path, state := range states { + if _, err := stmt.Exec(path, state.Size, state.MtimeNS); err != nil { + return fmt.Errorf("save scanned file %q: %w", path, err) + } + } + return tx.Commit() +} + // PendingEvents returns events due for upload at now, oldest first. A limit // of zero or less means no limit. func (s *DB) PendingEvents(now time.Time, limit int) ([]usage.Entry, error) { diff --git a/internal/usagescan/scanner.go b/internal/usagescan/scanner.go index ac68a95..8f70bc5 100644 --- a/internal/usagescan/scanner.go +++ b/internal/usagescan/scanner.go @@ -4,6 +4,7 @@ import ( "errors" "fmt" "log/slog" + "os" "path/filepath" "sort" "strings" @@ -80,6 +81,16 @@ func (s *Scanner) Scan(providerDirs map[usage.Provider][]string) (Result, error) var result Result var errs []error + // Stat snapshots of files already ingested. On error scan everything: + // re-parsing is wasted work, skipping is lost data. + scanned, err := s.db.ScannedFiles() + if err != nil { + scanned = nil + if s.Logger != nil { + s.Logger.Warn("scanned file states unavailable, scanning all files", "error", err) + } + } + for _, providerID := range s.scanOrder(providerDirs) { dirs := filterPaths(providerDirs[providerID]) if len(dirs) == 0 { @@ -90,7 +101,7 @@ func (s *Scanner) Scan(providerDirs map[usage.Provider][]string) (Result, error) errs = append(errs, fmt.Errorf("no usage provider registered for %q", providerID)) continue } - providerResult, err := s.scanProvider(provider, dirs) + providerResult, err := s.scanProvider(provider, dirs, scanned) result.setProviderResult(providerID, providerResult) if err != nil { errs = append(errs, err) @@ -100,9 +111,25 @@ func (s *Scanner) Scan(providerDirs map[usage.Provider][]string) (Result, error) return result, errors.Join(errs...) } -func (s *Scanner) scanProvider(provider usageprovider.Provider, paths []string) (ProviderResult, error) { +func (s *Scanner) scanProvider(provider usageprovider.Provider, paths []string, scanned map[string]usagedb.FileState) (ProviderResult, error) { var result ProviderResult - entries, err := providerWithPaths(provider, paths).Entries() + configured := providerWithPaths(provider, paths) + pending := make(map[string]usagedb.FileState) + if filterable, ok := configured.(filterConfiguredProvider); ok && scanned != nil { + configured = filterable.WithFileFilter(func(path string) bool { + info, err := os.Stat(path) + if err != nil || info.IsDir() { + return true + } + state := usagedb.FileState{Size: info.Size(), MtimeNS: info.ModTime().UnixNano()} + if previous, ok := scanned[path]; ok && previous == state { + return false + } + pending[path] = state + return true + }) + } + entries, err := configured.Entries() if err != nil { return result, err } @@ -111,6 +138,11 @@ func (s *Scanner) scanProvider(provider usageprovider.Provider, paths []string) if err != nil { return result, err } + // The stat snapshots were taken before parsing, so a write that lands + // mid-scan still changes the stored state and forces a re-scan. + if err := s.db.UpsertScannedFiles(pending); err != nil { + return result, err + } result.EventsParsed = len(entries) result.EventsInserted = inserted return result, nil @@ -182,6 +214,12 @@ type pathConfiguredProvider interface { WithPaths(paths []string) usageprovider.Provider } +// filterConfiguredProvider is implemented by providers that can skip source +// files the filter rejects. Providers without it are always fully scanned. +type filterConfiguredProvider interface { + WithFileFilter(filter usage.FileFilter) usageprovider.Provider +} + func providerWithPaths(provider usageprovider.Provider, paths []string) usageprovider.Provider { configured, ok := provider.(pathConfiguredProvider) if !ok { diff --git a/internal/usagescan/scanner_test.go b/internal/usagescan/scanner_test.go index 0bd0827..6602435 100644 --- a/internal/usagescan/scanner_test.go +++ b/internal/usagescan/scanner_test.go @@ -60,6 +60,77 @@ func TestScanInsertsBuiltInProviderEntries(t *testing.T) { } } +func TestScanSkipsUnchangedFiles(t *testing.T) { + dir := t.TempDir() + codexDir := filepath.Join(dir, "codex") + sessionDir := filepath.Join(codexDir, "sessions", "2026", "06", "04") + if err := os.MkdirAll(sessionDir, 0o700); err != nil { + t.Fatal(err) + } + sessionFile := filepath.Join(sessionDir, "rollout-session-a.jsonl") + if err := os.WriteFile( + sessionFile, + []byte( + `{"timestamp":"2026-06-04T01:02:03Z","type":"session_meta","payload":{"id":"session-a","cwd":"/Users/me/workspace/tokitoki"}}`+"\n"+ + `{"timestamp":"2026-06-04T01:02:04Z","type":"event_msg","payload":{"type":"token_count","info":{"last_token_usage":{"input_tokens":1,"output_tokens":2,"total_tokens":3}}}}`+"\n", + ), + 0o600, + ); err != nil { + t.Fatal(err) + } + db, err := usagedb.Open(filepath.Join(dir, "usage.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _ = db.Close() + }) + + scanner := New(db) + dirs := map[usage.Provider][]string{usage.ProviderCodex: {codexDir}} + + result, err := scanner.Scan(dirs) + if err != nil { + t.Fatal(err) + } + if parsed := result.Providers[usage.ProviderCodex].EventsParsed; parsed != 1 { + t.Fatalf("first events parsed = %d, want 1", parsed) + } + + result, err = scanner.Scan(dirs) + if err != nil { + t.Fatal(err) + } + if parsed := result.Providers[usage.ProviderCodex].EventsParsed; parsed != 0 { + t.Fatalf("unchanged file events parsed = %d, want 0", parsed) + } + + appended, err := os.OpenFile(sessionFile, os.O_APPEND|os.O_WRONLY, 0o600) + if err != nil { + t.Fatal(err) + } + if _, err := appended.WriteString( + `{"timestamp":"2026-06-04T01:02:05Z","type":"event_msg","payload":{"type":"token_count","info":{"last_token_usage":{"input_tokens":4,"output_tokens":5,"total_tokens":9}}}}` + "\n", + ); err != nil { + t.Fatal(err) + } + if err := appended.Close(); err != nil { + t.Fatal(err) + } + + result, err = scanner.Scan(dirs) + if err != nil { + t.Fatal(err) + } + codexResult := result.Providers[usage.ProviderCodex] + if codexResult.EventsParsed != 2 { + t.Fatalf("appended file events parsed = %d, want 2", codexResult.EventsParsed) + } + if codexResult.EventsInserted != 1 { + t.Fatalf("appended file events inserted = %d, want 1", codexResult.EventsInserted) + } +} + func TestScanUsesRegisteredProvider(t *testing.T) { dir := t.TempDir() db, err := usagedb.Open(filepath.Join(dir, "usage.db")) diff --git a/internal/usageupload/uploader.go b/internal/usageupload/uploader.go index 2b91db0..838d1c6 100644 --- a/internal/usageupload/uploader.go +++ b/internal/usageupload/uploader.go @@ -46,11 +46,13 @@ type Payload struct { Events []Event `json:"events"` } +// DevicePayload labels which machine a batch came from. It is display metadata +// only — event identity is the content hash in Event.ID, and the server dedupes +// on (user, event id). Nothing here may affect whether an event is stored. type DevicePayload struct { - InstallationID string `json:"installation_id"` - Name string `json:"name,omitempty"` - Platform string `json:"platform,omitempty"` - AppVersion string `json:"app_version,omitempty"` + Name string `json:"name,omitempty"` + Platform string `json:"platform,omitempty"` + AppVersion string `json:"app_version,omitempty"` } type Event struct { @@ -60,6 +62,17 @@ type Event struct { SourceProvider string `json:"source_provider,omitempty"` EventKind string `json:"event_kind,omitempty"` Timestamp string `json:"timestamp"` + // The machine's IANA zone ("Asia/Tokyo"), omitted when it cannot be + // resolved — see usage.MachineTimezone. Never a fixed abbreviation like + // "JST": those are ambiguous across regions and cannot be re-expanded. + // + // The UTC offset is deliberately not sent alongside it. Timestamp is an + // absolute instant and this is a zone, so the offset at that instant — + // including whether DST was in effect, and half-hour zones like + // Australia/Lord_Howe — is a pure function of the two. Sending it as well + // would be a second copy of a derived value, and the copy is what goes + // stale when tzdata is corrected. + Timezone string `json:"timezone,omitempty"` SessionID string `json:"session_id,omitempty"` Project string `json:"project"` ProjectPathHash string `json:"project_path_hash,omitempty"` @@ -168,18 +181,17 @@ func uploadBatch(ctx context.Context, settings agent.Settings, events []usage.En payload := Payload{ BatchID: "usage-" + time.Now().UTC().Format("20060102T150405.000000000Z"), Device: DevicePayload{ - // The server keys device rows on installation_id; an empty one - // (possible only for callers that hand-build Settings) falls back - // to a shared identity server-side. - InstallationID: settings.InstallationID, - Name: deviceName(), - Platform: usage.NormalizeOS(runtime.GOOS), - AppVersion: buildinfo.Resolved(), + Name: deviceName(), + Platform: usage.NormalizeOS(runtime.GOOS), + AppVersion: buildinfo.Resolved(), }, Events: make([]Event, 0, len(events)), } + // A property of the machine, not of any one event, so it is resolved once + // per upload rather than per event. + zoneName := usage.MachineTimezone() for _, entry := range events { - payload.Events = append(payload.Events, convertEvent(entry)) + payload.Events = append(payload.Events, convertEvent(entry, zoneName)) } body, err := json.Marshal(payload) @@ -219,7 +231,7 @@ func uploadEndpoint() string { return BaseURL() + "/api/usage-events/batch" } -// BaseURL is the TokiToki server every subsystem talks to — usage uploads and +// BaseURL is the Tokitoki server every subsystem talks to — usage uploads and // update checks alike. TOKITOKI_BASE_URL overrides the default. func BaseURL() string { value := strings.TrimRight(strings.TrimSpace(os.Getenv(BaseURLEnv)), "/") @@ -229,7 +241,21 @@ func BaseURL() string { return value } -func convertEvent(entry usage.Entry) Event { +// convertEvent maps one loaded entry onto the wire format. +// +// `zoneName` is this machine's IANA zone, or "" when it could not be resolved. +// It is passed in rather than looked up here so the lookup happens once per +// upload instead of once per event. +// +// A caveat worth stating plainly, because the field name cannot: the agent logs +// this reads store their timestamps in UTC, so the zone an event was *recorded* +// in is not recoverable from them. What is reported is the zone of the machine +// doing the reading. For the ordinary case — you code and upload on the same +// laptop — those are the same thing. For a backfill of logs copied from another +// machine, or from a machine you have since moved, they are not, and the server +// should treat this as "where the user was, approximately" rather than as an +// exact per-event fact. +func convertEvent(entry usage.Entry, zoneName string) Event { return Event{ ID: entry.ID, Provider: string(entry.Provider), @@ -237,6 +263,7 @@ func convertEvent(entry usage.Entry) Event { SourceProvider: string(entry.Provider), EventKind: entry.EventKind, Timestamp: entry.Timestamp.UTC().Format(time.RFC3339Nano), + Timezone: zoneName, SessionID: entry.SessionID, Project: entry.Project, ProjectPathHash: hashProjectPath(entry.ProjectPath), diff --git a/pkg/agentlib/agentlib.go b/pkg/agentlib/agentlib.go index 14ff47d..4ce5891 100644 --- a/pkg/agentlib/agentlib.go +++ b/pkg/agentlib/agentlib.go @@ -1,4 +1,4 @@ -// Package agentlib exposes TokiToki's local usage sync engine for native +// Package agentlib exposes Tokitoki's local usage sync engine for native // front-ends. package agentlib @@ -31,7 +31,7 @@ const ( // DefaultUploadTimeout is the maximum duration for one scan and upload run. DefaultUploadTimeout = 2 * time.Minute - // DefaultLockTimeout is the maximum duration to wait for another TokiToki + // DefaultLockTimeout is the maximum duration to wait for another Tokitoki // command to release the shared local data lock. DefaultLockTimeout = DefaultUploadTimeout + 10*time.Second ) @@ -118,7 +118,7 @@ type SyncOptions struct { ProviderDirs map[Provider][]string } -// Heartbeat describes one heartbeat-style IDE activity sample. +// Heartbeat describes one IDE activity sample. type Heartbeat struct { Entity string Timestamp time.Time @@ -204,7 +204,7 @@ func (c *Client) GetAPIKey() (string, error) { return settings.APIKey, nil } -// BaseURL returns the TokiToki server every subsystem talks to — usage +// BaseURL returns the Tokitoki server every subsystem talks to — usage // uploads, update checks, and the web dashboard alike. Front-ends open it // when they need a plain link to the server (for example as the fallback // when DashboardURL cannot mint a signed login link). @@ -434,7 +434,7 @@ func (c *Client) withUploadLock(fn func() error) error { return fn() } -// DefaultDataDir returns the shared TokiToki data directory. +// DefaultDataDir returns the shared Tokitoki data directory. func DefaultDataDir() (string, error) { home, err := os.UserHomeDir() if err != nil { diff --git a/scripts/gen-windows-zones.py b/scripts/gen-windows-zones.py new file mode 100644 index 0000000..bd74dac --- /dev/null +++ b/scripts/gen-windows-zones.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Regenerate internal/usage/windows_zones.go from the Go toolchain's own data. + +Go ships a Windows-zone table (zoneinfo_abbrs_windows.go) generated from the +Unicode CLDR windowsZones.xml, with the IANA name in a trailing comment. Reading +it here keeps the mapping in step with whatever Go version builds the CLI, +rather than vendoring a second copy of CLDR that would drift. + +Run after a Go upgrade: python3 scripts/gen-windows-zones.py +""" +import os +import re +import subprocess + +goroot = subprocess.run(["go", "env", "GOROOT"], capture_output=True, text=True).stdout.strip() +source = os.path.join(goroot, "src/time/zoneinfo_abbrs_windows.go") +pairs = re.findall(r'"([^"]+)":\s*\{[^}]*\},\s*//\s*(\S+)', open(source).read()) +if not pairs: + raise SystemExit(f"no zone pairs found in {source}") + +lines = [ + "// Code generated from Go's zoneinfo_abbrs_windows.go (itself generated from", + "// the Unicode CLDR windowsZones.xml). DO NOT EDIT.", + "// Regenerate with: python3 scripts/gen-windows-zones.py", + "", + "package usage", + "", + "// windowsToIANA maps a Windows time zone key name, as stored in the registry", + "// under HKLM\\SYSTEM\\CurrentControlSet\\Control\\TimeZoneInformation, to the", + "// IANA name the rest of the system speaks.", + "//", + "// Windows does not use IANA zones; it keeps its own list (\"Tokyo Standard", + "// Time\"), and the correspondence is maintained by the Unicode CLDR project.", + "// This table is the only way to obtain an IANA name on Windows: Go's", + "// time.Local reports the literal string \"Local\" on every platform, and the", + "// zone abbreviations it does expose (\"JST\") are ambiguous across regions.", + "var windowsToIANA = map[string]string{", +] +for windows_name, iana in sorted(pairs): + lines.append(f'\t"{windows_name}": "{iana}",') +lines.append("}") + +target = os.path.join(os.path.dirname(__file__), "..", "internal/usage/windows_zones.go") +with open(target, "w") as handle: + handle.write("\n".join(lines) + "\n") +subprocess.run(["gofmt", "-w", target], check=True) +print(f"wrote {len(pairs)} mappings to internal/usage/windows_zones.go")