Skip to content

FEAT expose mask parameter on OpenAIImageTarget for inpainting support #2006

@romanlutz

Description

@romanlutz

Summary

OpenAIImageTarget._send_edit_request_async currently calls client.images.edit(**image_edit_args) with only image, prompt, size, and a few formatting flags. The OpenAI SDK's images.edit endpoint also accepts an optional mask parameter — a PNG whose fully-transparent regions indicate the only pixels the model is permitted to modify, while every other pixel must be returned unchanged. This is inpainting, and it's the standard primitive for "keep this scene but regenerate just the masked subregion."

PyRIT does not expose mask today, so red-teamers using OpenAIImageTarget for image-editing attacks can only do whole-image edits. They cannot pin which subregion of the input image is allowed to change, which loses an important degree of control for evaluations that depend on precisely localized edits.

Proposed change

  1. Add a new MessagePiece data type or convention for image masks (options: a dedicated image_mask data type, or reuse image_path with a metadata flag — e.g. prompt_metadata={"is_mask": True}).
  2. In OpenAIImageTarget._send_edit_request_async, separate the input pieces into base image(s) and mask. When a mask is present, add "mask": (name, bytes, mime_type) to image_edit_args.
  3. Validate that mask is only attached when exactly one base image is provided (the OpenAI endpoint only supports masking a single base image — multi-image edits with a mask would need to be rejected with a clear error).
  4. Update the multimodal/router components (e.g. ModalityFeedbackRouter.build_objective_input_message) so attacks that want to forward a mask alongside the adversarial-generated text + base image can do so without rebuilding the message from scratch.
  5. Tests:
    • Mask piece is correctly serialized and passed through to the SDK call.
    • Multi-image + mask raises a validation error.
    • Backward compatibility: existing single-image and multi-image edit flows (no mask) continue to work unchanged.

API reference

OpenAI image edit endpoint: https://platform.openai.com/docs/api-reference/images/createEdit — see the mask parameter.

Scope / non-goals

  • No automatic mask generation from text (e.g. "mask the face") — masks are provided by the caller.

Acceptance criteria

  • A caller can construct a Message with [text, image_path, mask] and OpenAIImageTarget.send_prompt_async routes the mask correctly to client.images.edit(..., mask=...).
  • The unmasked portions of the input image are pixel-for-pixel preserved (verified in an integration test against a fixture image).
  • Existing tests continue to pass.

Metadata

Metadata

Assignees

Labels

not ready yetThis issue needs more definition or is blocked by a pending change.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions