AI Assistant

[yuripourre]

Hey @julien-moreau, I am creating a plugin to add an AI assistant to the editor, I think sometimes is just faster to ask an LLM instead of doing myself.

It would be a new tab in the editor with a text box. Initially I am pointing to my local ollama but would be possible to point to an external endpoint (with API key, etc.)

Currently I am not generating any texture or 3d models from the prompts but that's my idea.

What do you think about that?

[julien-moreau]

What would be awesome, would be that we discuss this on the forum to have ideas from others (https://forum.babylonjs.com)

I already started an expriment on my side which is on this branch: https://github.com/BabylonJS/Editor/tree/feat/ai-experiment.
The goal of this experiment is, by drag'n'dropping an image asset somewhere, to create a 3d model from the image using Hunyuan https://hunyuan-3d.com/

Here is a realtime video demo: https://babylonjs-editor.fra1.cdn.digitaloceanspaces.com/experiments/ai/hunyuan3d-2.mp4
Everything runs in local in a container using a RTX 5090 Laptop

[yuripourre]

@julien-moreau that's really cool!!!

Even more interesting because you are running it locally.

[Soullnik]

@yuripourre I think you should take all the best from @julien-moreau work and implement his idea as a plugin, I also think it is very important not just to add models to the scene but to be able to export them, this would be a super powerful feature for the editor, Also, for gamedev, a very cool feature would be the ability to create low-poly models.

[julien-moreau]

@Soullnik when you say "be able to export them", do you mean something like right-click the node in the graph and select "Export as GLB..." for example?

When you say "Also, for gamedev, a very cool feature would be the ability to create low-poly models.", can you elaborate a bit? It's interesting

@yuripourre I'm gonna push on the branch by adding a README for hunyuan3d (which is the only one that works today). Do you have a CUDA compatible machine? It requires CUDA to generate meshes (or it'll take ages to generate) and it's mandatory to generate textures

[Soullnik]

@julien-moreau As for export, you got it right, since we can't always ensure that the generated model fully meets our expectations, and it needs to be refined. Regarding low poly, usually 3D AI generators create very polygonal models by default, which affects the performance, and perhaps you can add a low poly setting in the AI plugin so that it creates low poly models. For example, I noticed that on average, without a drop in fps on standard hardware, no more than 200k polygons are required, and it would be cool to get models with an optimal number of polygons/quality

[yuripourre]

@yuripourre I'm gonna push on the branch by adding a README for hunyuan3d (which is the only one that works today). Do you have a CUDA compatible machine? It requires CUDA to generate meshes (or it'll take ages to generate) and it's mandatory to generate textures

I do, it can probably run huyuan I just need to figure out how to install it but should run without issues.

Yay, I am excited by thinking what the Editor can do with integrated AI.

[yuripourre]

This is what I had in mind #732

[kbpranay]

This sounds like a great idea for the BabylonJS editor! If you are looking for inspiration on how to handle the external endpoint/gateway logic (especially for handling tool-calling and different model backends like Claude or local Ollama), you might want to check out OpenClaw. It is an open-source, self-hosted gateway designed exactly for this kind of agentic workflow. Looking forward to seeing the plugin!