Show HN: Pip Imports in Deno(github.com)
github.com
Show HN: Pip Imports in Deno
https://github.com/denosaurs/deno_python
deno_python 0.3.1 adds support for importing python pip packages directly in JavaScript! Fun and useful, slightly cursed.
45 comments
The organization name "denosaurs" very roughly works for a combination of deno and python too: snakes fall into the clade of Lepidosaurs. Dinos are Archosaurs.
I wonder if there's another programming language featuring a reptile. What's this "Lizard" language Duckduckgo's telling me about?
I wonder if there's another programming language featuring a reptile. What's this "Lizard" language Duckduckgo's telling me about?
Maybe not to the same degree, but Zig!
Its users are even called Ziguanas.
Its users are even called Ziguanas.
Xahlee features a lizard for Common Lisp on his webpage:
http://xahlee.info/UnixResource_dir/writ/logo_lisp.html
http://xahlee.info/UnixResource_dir/writ/logo_lisp.html
Cobra
https://en.m.wikipedia.org/wiki/Cobra_(programming_language)
And pushing it a bit as "a language":
Anaconda
https://en.m.wikipedia.org/wiki/Cobra_(programming_language)
And pushing it a bit as "a language":
Anaconda
I always get a kick out of well named projects - good eye!
This is awesome and the right level of cursed!
Reminds me of PythonMonkey, which provides really high level bindings to SpiderMonkey in Python. It ships as both a pip package and a standalone js runtime, the latter of which is very similar to this project!
https://github.com/Distributive-Network/PythonMonkey#pythonm...
Reminds me of PythonMonkey, which provides really high level bindings to SpiderMonkey in Python. It ships as both a pip package and a standalone js runtime, the latter of which is very similar to this project!
https://github.com/Distributive-Network/PythonMonkey#pythonm...
Connecting these two would be even more cursed
Super cool! Fun, useful and slightly cursed indeed
An alternative is metacall. The example in the readme is about calling Python from Javascript, but it also works with other languages, like Ruby, C#, Java, and other languages
https://github.com/metacall/core
List of supported languages here https://github.com/metacall/core/blob/develop/docs/README.md...
In the future, maybe webidl (or extensions of it) will bring interoperability between languages too. At the moment there is https://mozilla.github.io/uniffi-rs/ for interoperability between Rust and a number of languages (basically the ones mozilla needs: Swift, Kotlin, Javascript)
https://github.com/metacall/core
List of supported languages here https://github.com/metacall/core/blob/develop/docs/README.md...
In the future, maybe webidl (or extensions of it) will bring interoperability between languages too. At the moment there is https://mozilla.github.io/uniffi-rs/ for interoperability between Rust and a number of languages (basically the ones mozilla needs: Swift, Kotlin, Javascript)
Let Python from Deno write to SQLite, Deno serve a webpage that use JS to accessing «static» SQLite using HTTP range requests, and we have come full circle
(Joke, but would be a cool prototype)
(Joke, but would be a cool prototype)
It would be magical if we also got typescript type hints too.
That sounds like a completely different beast of a tool
Not so much you can use inspect [0] to see a whole lot of information. I used this at work to auto generate json scheme of passed functions for openAI function feature [1]
[0] https://docs.python.org/3/library/inspect.html
[1] https://platform.openai.com/docs/guides/gpt/function-calling
[0] https://docs.python.org/3/library/inspect.html
[1] https://platform.openai.com/docs/guides/gpt/function-calling
Does this work for packages like PyTorch too, in GPU mode?
Yes, it uses your native python installation so everything that works with python should work with this.
I have had success running sbert, faster-whisper and various other GPU accelerated packages.
I have had success running sbert, faster-whisper and various other GPU accelerated packages.
Even though this is a but cursed, it could allow more people to access Pythons large and exclusive ecosystem libraries!
This is really fun indeed, well done, I like it. It’s funny how you decided to demo it with a library that does lots of c++ bindings it’s self.
Now someone can put a wrapper around this to hide the python imports, so we can claim JS is the only language we need.
You can also do this in Nim, which basically means you can write any program you could in Python with libraries in Nim. https://github.com/yglukhov/nimpy
Is there an equivalent for Node? I'm glad I haven't needed it since, but I couldn't find anything nearly so seamless when I last looked.
why can't you just exec a python shell command to do the same thing? i'm actually slightly puzzled that "FFI" is its own thing when every language has a way to exec shell commands
deno_python exposes Python modules to JavaScript in a way that makes it possible access the API of any Python module out of the box, as if it were a JS API. Executing a Python shell command is quite different.
Here, FFI is used to interface with the Python Interpreter C API. Argument for FFI against just executing shell commands is performance, efficiency, and that you can make calls from same process instead of spawning a subprocess. When doing [de]serialization over shell interface, a lot of things will be hit and miss. But for deno_python, type conversions are widely supported. You can even pass around objects by reference with zero copy, which would otherwise require copy over a shell interface.
Here, FFI is used to interface with the Python Interpreter C API. Argument for FFI against just executing shell commands is performance, efficiency, and that you can make calls from same process instead of spawning a subprocess. When doing [de]serialization over shell interface, a lot of things will be hit and miss. But for deno_python, type conversions are widely supported. You can even pass around objects by reference with zero copy, which would otherwise require copy over a shell interface.
You can if you are just executing a static python script never changes.
But if you want to call some python functions with your TS input then you would need to serialize the TS input, pass it as an arg to the script, there deserialize it and then call the python function with the input. And if you wanted to call multiple python functions then your first arg to the python script would need to be the name of the function, only then the serialized data. Plus you would need to implement a switch case on the python side to execute the right function.
But if you want to call some python functions with your TS input then you would need to serialize the TS input, pass it as an arg to the script, there deserialize it and then call the python function with the input. And if you wanted to call multiple python functions then your first arg to the python script would need to be the name of the function, only then the serialized data. Plus you would need to implement a switch case on the python side to execute the right function.
right. ok. thats not hard^? like at all? and in fact ive had multiple bugs that ultimately had to do with serialization being quietly abstracted over for me that if i had just handrolled it myself i probably wouldve been consciously aware of my bad choices.
^i know this is probably arrogance out of naivete, just teasing out what exactly i dont know that i dont know about what FFI does.
^i know this is probably arrogance out of naivete, just teasing out what exactly i dont know that i dont know about what FFI does.
There are limits to what the commandline arguments can take. And even if you fallback to files instead, you still need to serialise a potentially huge amount of data. (Wanna serialise a 1GB tree that is going to get only one lookup in each call to the script?) On top of that, you're missing any chance of in-memory caching, because the script call can't preserve that, while FFI could. Then there are things that are just not possible to serialise like pointers to mapped memory, file descriptors (outside of unix sockets), etc.
Finally even if everything works, a command execution and parsing results is massively slower than a function call. So yeah, there's lots of restrictions on what you can practically do that way.
Finally even if everything works, a command execution and parsing results is massively slower than a function call. So yeah, there's lots of restrictions on what you can practically do that way.
So you can now shoot yourself in the foot with both Python and JavaScript package management?
Deno fixed the mess of JavaScript package management... But even NPM is light years ahead of Python. I have no idea why you'd want to inflict that on yourself once you've escaped it. I guess the only reason is there's some Python package that you have no option but to use.
The package targets deno. It can be imported directly without using any package manager.
So you can shoot just one foot with pip
Javascript's package management is so much better than Python's IMO...
I believe that's the joke, the two package systems have orthogonal nightmares
I had missed this message but have to say I love the expression "orthogonal nightmares", really captures it well :)
Package management, sure. The packages themselves leave a lot to be desired in JS.
Everyday we stray further from gods light.
I'm using this package so far.
https://github.com/hmenyus/node-calls-python
Excited to compare to Deno. Thanks guys!
https://github.com/hmenyus/node-calls-python
Excited to compare to Deno. Thanks guys!
Next step would be getting ObservableJS running and using it to write notebooks. (There is a vscode plugin for this that might be adapted.)
Is Deno really performing so badly that it has to get PIP'ed? I thought NodeJs would be first...
Not at all but if you have an api in js and you want to do any sort of machine learning, ai, forecasting well all the best tools are in python.
The you have two choices you rewrite your api into python or you call python from your js.
The you have two choices you rewrite your api into python or you call python from your js.
A truly incredible feat of FFI!
ngl this is a little cursed, but awesome
Nice !