feat(llmobs): accept image_parts on messages - #9684
Conversation
Adds image support to the LLM Observability SDK, mirroring audio_parts. A
message may carry imageParts, each `{mimeType, content | attachmentKey}`, which
the tagger validates and emits as the snake_case wire shape `image_parts:
[{mime_type, content | attachment_key}]` — the same shape dd-trace-py emits and
the backend already types.
formatAudioPart and formatImagePart share one builder, since audio and image
parts have an identical wire shape and the linter rejects the duplicate.
Manual annotation only; provider auto-capture is a follow-up.
Overall package sizeSelf size: 7.85 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🔄 Datadog retried 1 test - 1 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: f555ba2 | Docs | Datadog PR Page | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9684 +/- ##
=======================================
Coverage 98.51% 98.51%
=======================================
Files 963 963
Lines 135796 135858 +62
Branches 11984 11925 -59
=======================================
+ Hits 133785 133847 +62
Misses 2011 2011
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
BenchmarksBenchmark execution time: 2026-08-05 19:35:13 Comparing candidate commit f555ba2 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2316 metrics, 42 unstable metrics.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54323e6397
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| /** | ||
| * Key of an already-uploaded image, in place of inline content | ||
| */ | ||
| attachmentKey?: string, |
There was a problem hiding this comment.
For callers using the new TypeScript surface, the runtime rejects an image part when neither payload is set or when both content and attachmentKey are set, but the declaration makes both fields optional, so TypeScript accepts both invalid shapes and users only find out at runtime when llmobs.annotate() throws. Please model ImagePart as a union requiring exactly one payload field so the public type matches the validation contract.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Done! Updated ImagePart to a union, matching the existing AssistantTextMessage pattern.
|
Tried this branch out end to end on a real app — it works. Wiring up a vision step in the JS stock watchlist test app (DataDog/llm-observability#91) and the input image renders on the span in the trace view. Setup # in test-apps/stock-watchlist-agent-js
npm install ../../../dd-trace-js # this branch, jose/mlob-7916-llmobs-image-parts
DD_LLMOBS_ML_APP=stock-watchlist-agent-js-chris dd-auth -- npm start -- logos/apple.pngThe app takes a logo image, calls an OpenAI vision model to translate it into a ticker symbol, then runs its normal research flow on that ticker. The annotation on the vision span: llmobs.annotate(span, {
inputData: [
{ role: 'system', content: VISION_PROMPT },
{ role: 'user', content: 'Which publicly traded company does this image show?',
imageParts: [{ mimeType: 'image/png', content: base64 }] },
],
})The image shows up on the Two things worth calling out, both about discoverability rather than correctness:
Nothing blocking from my side. |
Address review feedback on the public typing surface.
index.d.v5.ts now declares Message.imageParts and ImagePart. AGENTS.md
requires a new public type in both files unless the API is v6-only, and
this one is not: the runtime backports and audioParts already ships in
v5. No tsconfig references index.d.v5.ts, so it was verified by compiling
that surface standalone and resolving llmobs.ImagePart against it.
ImagePart becomes an exclusive union carrying exactly one of content or
attachmentKey, using the "?: never" shape already used by
AssistantTextMessage and AssistantToolCallMessage in the same file.
docs/test.ts pins all four cases, two valid and two behind
ts-expect-error. Those assertions are load-bearing: reverting the type to
all-optional fields fails type:doc:test with TS2578 twice.
The union is enforced on a directly annotated ImagePart but not on an
inline literal passed to annotate(), since inputData and outputData
include a "{ [key: string]: any }" arm that disables excess-property
checking. Narrowing that affects every annotate() shape and is left out.
Tests: the image non-string-content case now asserts the
invalid_io_messages telemetry tag that its audio counterpart already
asserted, closing a hole where deleting the tag argument kept the suite
green. An SDK-level image test mirrors the audio one, and three image
test names are aligned to the audio wording.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 032e10eb9a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| /** | ||
| * Images attached to the message (e.g. vision input, generated output) | ||
| */ | ||
| imageParts?: ImagePart[], |
There was a problem hiding this comment.
Fresh evidence after the ImagePart union change: actual llmobs.annotate({ inputData: ... }) calls with imageParts: [{ mimeType: 'image/png' }] or with both content and attachmentKey still type-check, because AnnotationOptions.inputData/outputData can fall through to the broad { [key: string]: any } union arm instead of this Message field. That means the public API still accepts the shapes the tagger now throws on at runtime; please tighten the annotation input types (and mirror the v5 surface) so object-literal annotate calls get the same exclusivity check.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed. inputData and outputData use an index signature ({ [key: string]: any }) that disables excess-property checks, matching existing AudioPart behavior. Narrowing this index signature impacts all annotate() shapes, so it is best handled in a separate PR to keep this change purely additive.
| /** | ||
| * Images attached to the message (e.g. vision input, generated output) | ||
| */ | ||
| imageParts?: ImagePart[], |
There was a problem hiding this comment.
When a span processor is registered and an LLM span is annotated with imageParts, span_processor.js copies the tagger's normalized messages through to LLMObservabilitySpan.input/output, so processors receive messages containing image_parts. The public processor type still exposes only { content, role? }[], so TypeScript processors that need to inspect or redact image payloads (for example span.input[0].image_parts) fail to compile even though the runtime now sends that field; please add the processor message shape for image parts in both public declaration files.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed. LLMObservabilitySpan.input also omits audio_parts, tool_calls, and tool_results sent at runtime. I will open a follow-up PR to update the processor message type across all media and tool fields together rather than special-casing images.
| /** | ||
| * Images attached to the message (e.g. vision input, generated output) | ||
| */ | ||
| imageParts?: ImagePart[], |
There was a problem hiding this comment.
Because Prompt.template is typed as string | Message[], adding imageParts to Message now lets TypeScript callers put images in prompt chat templates, but tagPrompt() serializes those templates with only { role, content }, so the image parts are silently dropped from meta.input.prompt.chat_template. In code that annotates a prompt template alongside image messages, users get a clean type-check but no recorded images; please either include image parts in the prompt wire shape or split the prompt-template message type to exclude media, mirroring the chosen public type change to v5 as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed. This is pre-existing behavior shared with audioParts, toolCalls, and toolResults. I will address prompt wire shapes for all media/tool fields in a dedicated follow-up PR to avoid scope creep here.
sabrenner
left a comment
sabrenner
left a comment
There was a problem hiding this comment.
i think the new / remaining codex comments are OK to be left unaddressed here and are applicable potentially in follow ups
3 participants
What does this PR do?
Adds image support to the LLM Observability SDK. A message may carry
imageParts, each{ mimeType, content | attachmentKey }, which the tagger validates and emits as the snake_case wire shapeimage_parts: [{ mime_type, content | attachment_key }].Message.imagePartsand anImageParttype are declared in both public typing surfaces,index.d.tsandindex.d.v5.ts.ImagePartis an exclusive union, so a part carrying neithercontentnorattachmentKey, or both at once, fails to type-check rather than only throwing at runtime.docs/test.tspins all four cases, two valid and two behind@ts-expect-error.Manual annotation only: this is
llmobs.annotate(), and an auto-instrumented OpenAI or Bedrock call that sends an image will not populateimage_partsby itself. Provider auto-capture is a follow-up.Motivation
This is the wire shape
dd-trace-pyalready emits and the backend already types, so JS was the gap: an application could send images to a model but could not record them on the span. It mirrorsaudioParts, which landed earlier by the same route.Additional Notes
The change is purely additive: 326 insertions across 6 files, no deletions, and no existing code path modified.
#filterImagePartsis a new private method; the only edits to existing functions are one destructured field and one guarded emit block in#tagMessages.Validation rejects a non-object part, a missing or empty
mimeType, neithercontentnorattachmentKey, both at once, and a non-string value for either. Each failure is taggedinvalid_io_messagesfor telemetry, matching the audio path. An all-invalid list omitsimage_partsrather than emitting an empty array.Three things deliberately left out of scope:
formatImagePartwas in an earlier revision of this branch but had no caller, so it is dropped here and will land with the instrumentation that uses it. That also leaves the sharedutil.jsuntouched by this PR. Note this differs from the audio precedent, where feat(llmobs): capture audio in messages and OpenAI chat completions #9083 shipped the tagger and the OpenAI capture together.AudioPartinindex.d.tsdeclares onlycontent, though the tagger also acceptsattachmentKeyfor audio. That is a pre-existing gap on the audio side and is not corrected here.llmspan kinds. OnlyspanKind === 'llm'routes totagLLMIO(sdk.js:278), so on atask,workfloworagentspan animagePartsarray falls through totagTextIOand is stringified whole, with no validation failure and no telemetry. Pre-existing behaviour shared withaudioParts, and worth its ownfix(llmobs).npm run lintis clean.npm run type:checkis not clean on this branch, but nothing added here is responsible:index.d.tsreports zero errors, and the 18 intagger.jsall sit between lines 109 and 495, none of them in the regions this PR adds (699-751, 779, 821-828).Two limits on what the type tooling proves, stated rather than implied.
index.d.v5.tsis referenced by notsconfig, so no CI job type-checks it; it was verified by compiling that surface standalone. And the@ts-expect-errorassertions cover a directly annotatedImagePartbut not theannotate()call site, sinceinputDataandoutputDatainclude a{ [key: string]: any }arm (index.d.ts:4136,:4144) that disables excess-property checking.How to test
No credentials needed. Validation and wire shape are covered by unit tests:
End to end, if you want to see it render. Manual annotation needs no LLM provider key, only a Datadog key:
On
masterthat same call records nothing.#tagMessagesdestructures a fixed key list ending ataudioParts(tagger.js:725) and there is no unknown-key warning, soimagePartsis dropped with no error and no telemetry. Silent data loss is the behaviour this fixes.tagger.jsis the only runtime file changed here (the other five are type definitions and tests), so a before/after is a one-file swap against the same install:Before / after