feat(data-warehouse): implement openai import source#70722
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.79 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.1 KiB | ../node_modules/.pnpm/posthog-js@1.401.0/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.7 KiB | ../node_modules/.pnpm/posthog-js@1.401.0/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 106.1 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🔺 +1.6 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1308.69 MiB · 🔺 +1.6 KiB (+0.0%)
⚠️ Playwright — 1 failed
🎭 Playwright report · View test results →
❌ 1 failed test:
- password-protected insight sharing (chromium)
These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!
⚠️ Backend coverage — 96.0% of changed backend lines covered — 19 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ███████████████████░ 96.0% (562 / 581)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/openai/source.py |
90.6% | 114–115, 117 |
products/warehouse_sources/backend/temporal/data_imports/sources/openai/openai.py |
92.2% | 58–60, 67, 107, 167, 260, 324–326, 344–346, 358–360 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29427991659 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
█████████████░░░░░░░ 67.4% |
25,473 / 37,804 |
signals |
████████████████░░░░ 79.1% |
19,074 / 24,111 |
data_modeling |
████████████████░░░░ 80.0% |
4,834 / 6,045 |
cdp |
████████████████░░░░ 80.7% |
3,118 / 3,864 |
agent_platform |
█████████████████░░░ 84.2% |
3,112 / 3,695 |
notebooks |
█████████████████░░░ 84.3% |
6,323 / 7,501 |
cohorts |
█████████████████░░░ 86.0% |
3,989 / 4,639 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.3% |
6,891 / 7,800 |
conversations |
██████████████████░░ 88.9% |
16,126 / 18,130 |
dashboards |
██████████████████░░ 89.0% |
5,647 / 6,345 |
mcp_analytics |
██████████████████░░ 89.1% |
2,502 / 2,807 |
error_tracking |
██████████████████░░ 89.5% |
9,691 / 10,824 |
alerts |
██████████████████░░ 89.9% |
3,638 / 4,046 |
engineering_analytics |
██████████████████░░ 90.0% |
5,065 / 5,625 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
slack_app |
██████████████████░░ 90.6% |
9,511 / 10,503 |
marketing_analytics |
██████████████████░░ 90.8% |
11,514 / 12,684 |
product_analytics |
██████████████████░░ 91.1% |
5,527 / 6,068 |
data_warehouse |
██████████████████░░ 92.1% |
17,915 / 19,459 |
workflows |
██████████████████░░ 92.4% |
5,148 / 5,574 |
web_analytics |
███████████████████░ 92.7% |
13,607 / 14,674 |
ai_observability |
███████████████████░ 92.8% |
14,863 / 16,014 |
surveys |
███████████████████░ 92.9% |
5,660 / 6,094 |
posthog_ai |
███████████████████░ 93.2% |
1,312 / 1,408 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
skills |
███████████████████░ 94.4% |
2,827 / 2,995 |
revenue_analytics |
███████████████████░ 94.5% |
3,598 / 3,809 |
review_hog |
███████████████████░ 94.6% |
6,475 / 6,848 |
logs |
███████████████████░ 95.3% |
9,528 / 9,994 |
experiments |
███████████████████░ 95.6% |
24,138 / 25,244 |
replay_vision |
███████████████████░ 95.7% |
13,300 / 13,896 |
feature_flags |
███████████████████░ 96.1% |
14,750 / 15,354 |
warehouse_sources |
███████████████████░ 96.2% |
222,397 / 231,294 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
data_catalog |
███████████████████░ 97.1% |
2,034 / 2,095 |
customer_analytics |
███████████████████░ 97.2% |
7,480 / 7,698 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
StatusReview feedback — resolved. The project fan-out path was advancing the resume checkpoint to the next project while the current project's rows could still be buffered in the batcher (flushed only at the very end), so a crash in between would lose them on resume. Fixed by flushing incomplete batches before checkpointing, with a regression test ( CI — all real checks are green. The only red check is
These are all preview-provisioning infrastructure failures — the diff here is a data-warehouse backend source plus an Merge — the branch is conflict-free and up to date; the remaining 🦉 via talyn.dev |
Fill in the scaffolded OpenAI warehouse source: usage/costs bucketed endpoints, org/project entity tables, project fan-out, and audit logs, following the anthropic source pattern (ResumableSource + tracked HTTP). Generated-By: PostHog Code Task-Id: a318eafb-08f9-49ad-b05c-2868535919ad
…eckpoint In the project fan-out path, the resume state was advanced to the next project while the current project's rows could still be buffered in the batcher, which is only flushed at the very end. A crash between the state save and the final flush would drop those buffered rows on resume, since resume restarts at the checkpointed next project. Flush any incomplete batch before checkpointing to the next project, and add a regression test asserting a project's rows are yielded before the cursor advances. Generated-By: PostHog Code Task-Id: f647c7e7-718a-43ed-8283-01c68be8944c
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
Gilbert09
force-pushed
the
posthog-code/openai-warehouse-source
branch
from
85b456f to
54cb3f1
Compare
2 participants
Problem
OpenAI is the default LLM provider for most companies, so organization-level API spend and usage history is one of the most requested AI datasets to land in the warehouse. The
openaisource existed only as a scaffolded stub (registered, unreleased, no fields or sync logic).Changes
Implements the OpenAI import source end-to-end, following the same architecture as the Anthropic source (whose admin API closely mirrors OpenAI's):
settings.py– declarative endpoint catalog: 18 tables covering the Usage API (usage_completions,usage_embeddings,usage_moderations,usage_images,usage_audio_speeches,usage_audio_transcriptions,usage_vector_stores,usage_code_interpreter_sessions), the Costs API (costs), org dimension tables (projects,users,invites,admin_api_keys), project fan-out tables (project_users,project_service_accounts,project_api_keys,project_rate_limits), andaudit_logs.openai.py– transport: bearer-auth GETs viamake_tracked_session(), tenacity backoff on 429/5xx, page-token pagination for the bucketed usage/costs endpoints andafter-cursor pagination for entity lists, resumable state saved after each yielded batch.source.py–ResumableSourcewiring: single Admin API key field, static schema catalog (lists_tables_without_credentials), credential validation (401 rejects, 403 accepted at create time), non-retryable 401/403 messages.canonical_descriptions.py– table/column descriptions from the official API docs for all 18 tables, feeding semantic enrichment and the public docs table list.Sync semantics:
start_time(server-side Unix-seconds filter) with a trailing-day lookback, since OpenAI restates a day's bucket as late usage lands. Rows are one-per-(bucket × group-by combination) with a synthesized hashidover the identity dimensions only, so restated buckets merge in place.effective_atvia the server-sideeffective_at[gte]filter. They return newest-first with no order param, so the endpoint declaressort_mode="desc"and the watermark only commits at sync completion.Note
Conservative choices where live verification wasn't possible (I had no real Admin API key to probe with, endpoints answer 401 without one): bucket ordering for usage endpoints was verified oldest-first against the OpenAI cookbook's pagination example rather than a live call, the wire format for
group_by(repeated params) andeffective_at[gte](bracket style) matches the official SDK's query serialization, and the documented-in-the-wild "empty page with anext_pagetoken" quirk on/organization/costsis guarded by stopping pagination on an empty page.The source stays behind
unreleasedSource=TruewithreleaseStatus=ALPHAwhile it awaits a live-credential shakedown; flipping it visible is a one-line follow-up.User-facing doc: PostHog/posthog.com#18463 (
docsUrlhere matches its slug;audit_source_docspasses for this source against that checkout).How did you test this code?
Automated tests only (no live API credentials were used):
tests/test_openai.py(transport): bucket pagination follows/resumes page tokens and stops on the empty-page costs quirk (would otherwise loop forever); entity cursor pagination advances onafterand falls back to the last item id whenlast_idis absent (would otherwise silently truncate to one page); audit-log incremental runs send theeffective_at[gte]filter (without it every sync refetches all history); surrogate-id stability across metric restatements (a change would duplicate rows on every merge); owner flattening for both API key shapes; credential status mapping (200/403 valid, 401 invalid); retry budget for 429/5xx vs no retry for 4xx.tests/test_openai_source.py(source class): schema catalog completeness, incremental flags only on endpoints with genuine server-side filters, composite primary keys on fan-out children (a bareidkey would multi-match on merge and OOM),sort_mode="desc"on audit logs, documented-tables output for public docs.All 63 tests pass via
hogli test;ruff check/ruff formatandhogli ci:preflight --fixare clean.makemigrations --checkreports no changes (the enum already existed from scaffolding).Automatic notifications
Docs update
Doc added in PostHog/posthog.com#18463.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Authored with PostHog Code (Claude). Skills invoked:
/implementing-warehouse-sources,/documenting-warehouse-sources,/writing-tests. Decisions: modeled on the Anthropic source (OpenAI's admin API is its close analog) rather than the genericrest_source.RESTClient, because the bucketed usage endpoints need row synthesis and resumable page-token state that the sibling source already solves; kept the scope to Admin-API-key endpoints and left project-key datasets (batches, fine-tuning jobs, files) out, since they'd need a second credential and the headline datasets are org usage/spend; regeneratedgenerated_configs.pyand reverted an unrelated environment-dependent Stripe hunk the generator produced locally.Created with PostHog Code