Skip to content

Multimodal inputs in the DraftValidation class #394

Description

@howardbaik

Prework

Proposal

Continuing on the theme of multimodal inputs, I thought it'd be really cool if the DraftValidation class could support multimodal inputs (images and PDF), just as Validate.prompt() now does. A rough sketch of how that would work, drawing inspiration from the How DraftValidation Works steps:

  1. A statistical summary of your data is generated using the DataScan class.
  2. This summary is converted to JSON format and sent to your selected LLM provider.
  3. NEW: You provide the DraftValidation class with an image or PDF file describing what the data should look like. For example, the image could be a schema diagram, while the PDF could outline the project configuration elements that the data should conform to.
  4. The LLM uses the summary and multimodal input along with knowledge about Pointblank’s validation capabilities to generate a validation plan.
  5. The result is returned as executable Python code that you can use directly or modify as needed

Happy to get feedback from you @rich-iannone, and iterate. I'd love to implement this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions