Repository navigation
Ability to send a message from Python back to Elixir #20
Description
We have a few interesting use cases for Pythonx at TV Labs, and one of the features that would make it particularly useful, is if long-running Python programs could be executed, with the ability to send messages back to the owning process (or any process).
There is precedent for a feature like this, since NIFs themselves can send message through the enif_send interface.
@cocoa-xu is on vacation this week or I would have asked her, but I imagine this is really only possible if we are able to pull the BEAM header files into Python, or find a way to expose a function in Python that can lift the message up to the BEAM layer.
Thoughts?
Activity
josevalim commentedon Apr 10, 2025
Python can easily call C and this library ships with C code, so one option would be to define a C function alongside our C code and then call it from Python. However, in order to do that, you would probably need to store the environment somewhere (if we don't already do that).
josevalim commentedon Apr 10, 2025
And yes, we would need to expose this function somewhere. Perhaps this lib could ship with a .py file in priv, I am not sure of the best approach.
davydog187 commentedon Apr 10, 2025
Right, after looking at the implementation for a moment, there is a python prelude already in the NIF where the function could be exposed
https://github.com/livebook-dev/pythonx/blob/main/c_src/pythonx.cpp#L388
Following the example of
pythonx_handle_io_write, we could expose a function that is quite similar that does theenif_sendbut off of user input https://github.com/livebook-dev/pythonx/blob/main/c_src/pythonx.cpp#L1399I think I could throw a prototype together here
jonatanklosko commentedon Apr 10, 2025
Yeah, I think this could be done very similar to IO write.
We could patch the default globals to include an extra function that we would define, or rather a
pythonxpython module with such function, imported by default. We would do this as part of initialization, similarly to how we define custom stdout/stdin objects. The only issue is that this makes the Python code specific to Pythonx, but maybe it's fine for specific use cases.Note that with IO you can already send messages wherever you want by specifying custom
:stdout_device, although it only works with text. I assume you want to send a Python object as the message. One thing to note though is that once you get that Python object, any operations on that object will requires the GIL. If the long-running Python code doesn't hold GIL for much, then it should be fine, but if it does, it can result in bottlenecks.davydog187 commentedon Apr 10, 2025
That wasn't clear to me but is obvious in retrospect
Correct, although I think there are two modes that I might care about
My motivations here are more for eventing that I think is better suited to 2
josevalim commentedon Apr 10, 2025
The issue with 1 is precisely that you would trigger the
gilso you would have to be very careful about it. Perhaps one option is to allow only binaries to be sent...jonatanklosko commentedon Apr 10, 2025
For 2 we also need GIL, we don't really want to decode within the NIF itself, because that has limitations. In other words instead of 2. we would actually do 1. and you call
Pythonx.decode/1, so it's the same.josevalim commentedon Apr 10, 2025
What about 3, only send binaries which we already encode as binaries before sending them?
davydog187 commentedon Apr 10, 2025
@josevalim isn't that equivalent to setting the
:stdout_device, or do you mean something else?josevalim commentedon Apr 10, 2025
Yes, it would be equivalent.
josevalim commentedon Apr 10, 2025
Well, one slight difference, you would be to send it to different PIDs if we allowed PIDs to be encoded to Python.
davydog187 commentedon Apr 10, 2025
Right. I may try to find some time to prototype something here, will report back
josevalim commentedon Apr 10, 2025
But to be clear, if IO works for you, then go for it. That’s how you would communicate through a port anyway.
lalabuy948 commentedon May 12, 2025
Hi everyone, we potentially have a nice usecase for this library at CyanView, I'm working on PoC at the moment.
@davydog187 did you have a chance to make prototype for exchanging events between python and elixir?
josevalim commentedon May 12, 2025
Do you have an example of what you are trying to do? It is worth saying that you can emulate this by calling the Python code, having it stop once you have the intermedia result, yielding back to Elixir, and then you call it back passing the previous state/objects. This is probably even preferable to letting Python run in a dirty scheduler for really long periods of time.
lalabuy948 commentedon May 13, 2025
At the moment I work on integration with streamdeck and since we are running purely Elixir on our devices we would love to continue doing that, Python is being used to listen for "button pressed" events from the streamdeck and I would love to pass those events back to Elixir code, so it's a long running process. Worse case scenario I will pass those events as api calls back to my Elixir app.
davydog187 commentedon May 13, 2025
josevalim commentedon May 13, 2025
I see! I'm honestly not sure how I feel about having a whole Python application running forever within a dirty scheduler. Unless you are really doing something high performance that requires memory sharing, I think using something like
erlport(or similar) for I/O communication or an API would be a better fit. But maybe it would all be fine too, I just never tried it!davydog187 commentedon May 13, 2025
It's possible that erlport would be a better solution, but experimenting with Pythonx is simply too tempting
cocoa-xu commentedon May 15, 2025
I have a working POC for this, and we can figure out the details later -- like if the Python objects should be encoded as Erlang terms or passed as opaque handles, or just make it an option for the user to choose.
GenericJam commentedon Dec 19, 2025
Is there any progress here? I would love to be able to use this feature.
jonatanklosko commentedon Dec 21, 2025
See #38 :)
jonatanklosko commentedon Dec 22, 2025
We merged #38. You can try it by installing Pythonx from GitHub
{:pythonx, github: "livebook-dev/pythonx"}. Feedback is welcome :)