Building Standalone Python Applications with PyOxidizer(gregoryszorc.com)
gregoryszorc.com
Building Standalone Python Applications with PyOxidizer
https://gregoryszorc.com/blog/2019/06/24/building-standalone-python-applications-with-pyoxidizer/
59 comments
Isn't it glorious?
I definitely share those concerns. But being able to package things in an independent way is incredibly useful.
I hope we can get away from the reliance on incredibly disproportionately sized runtimes (such as Chromium) for making GUI apps in most cases. I think it trades too much of the user's resources for developer convenience. Unless you are trying to do audio/video conferencing in which case I guess I get it.
I've found the work on the Scenic UI framework in Elixir quite cool (only depends on OpenGL + glfw on desktop, nanovg on RPi).
I also had reason to build a simple bit with wxWidgets in Elixir through Erlang. Didn't love it. But that should be able to make reasonable-sized native but cross-platform UIs. Elixir and Erlang tend to package releases in a nice independent way.
But what I need this for is because every library under the sun is available for Python. And sometimes you need to just get a small utility built and distributed.
I definitely share those concerns. But being able to package things in an independent way is incredibly useful.
I hope we can get away from the reliance on incredibly disproportionately sized runtimes (such as Chromium) for making GUI apps in most cases. I think it trades too much of the user's resources for developer convenience. Unless you are trying to do audio/video conferencing in which case I guess I get it.
I've found the work on the Scenic UI framework in Elixir quite cool (only depends on OpenGL + glfw on desktop, nanovg on RPi).
I also had reason to build a simple bit with wxWidgets in Elixir through Erlang. Didn't love it. But that should be able to make reasonable-sized native but cross-platform UIs. Elixir and Erlang tend to package releases in a nice independent way.
But what I need this for is because every library under the sun is available for Python. And sometimes you need to just get a small utility built and distributed.
Disk is cheap, internet is pretty fast. I'd rather have the consistency of packaged software than deal with getting a bunch of libraries just right to save a little space.
Sadly, this attitude goes a long way towards explaining why all software is garbage and computers feel about the same speed now as they did in 1998, despite being literal orders of magnitude more powerful.
Maybe we'll eventually hit hard physical limits on hardware, and then we software developers will be forced to learn how to do our jobs correctly for the first time.
Maybe we'll eventually hit hard physical limits on hardware, and then we software developers will be forced to learn how to do our jobs correctly for the first time.
> Disk is cheap, internet is pretty fast
Until it isn't [0]. I think it's fine to only develop/care for faster hardware but I'm not a fan of broad statement like this when it doesn't apply for so many people.
[0] https://www.speedtest.net/global-index#mobile
Until it isn't [0]. I think it's fine to only develop/care for faster hardware but I'm not a fan of broad statement like this when it doesn't apply for so many people.
[0] https://www.speedtest.net/global-index#mobile
There is a difference between catering to fast runtime internet vs fast install internet. I agree with you that we should optimize apps to be quick on slow internet, especially with so many mobile only users. But making an app that is large is a one time hit that isn't so bad, even on slow internet.
Bold of you to assume there will be 47 useful apps in 2025
I'm assuming 45 of them will be pre-installed by the OEM and run in the background using Electron-for-daemons(tm) to enable a more personalized advertorial experience.
I care that my applications work correctly. I don't care what is under the hood. I care that it works, correctly and well.
If anything, the technology underlying the abstraction of data storage should be taking care of optimal use of resources.
If anything, the technology underlying the abstraction of data storage should be taking care of optimal use of resources.
We used to call that an operating system.
Update 2: Gave up on MSYS2 and just used rustup. Managed to compile and run, but exit() doesn't work. Also I just realized the executable contains the compilation path, which leaks private information? Makes it hard to compile and redistribute stuff, but otherwise I'd be very excited about the project.
__________________________________________________
Update: Still can't get it to work. Also, the tool seems to download random third-party packages that aren't documented?
I ran 'cargo install pyoxidizer' and got
Edit: Would be nice if someone could also explain how this is a manifest parsing error?
__________________________________________________
Update: Still can't get it to work. Also, the tool seems to download random third-party packages that aren't documented?
> pyoxidizer run
no existing PyOxidizer artifacts found
processing config file pyoxidizer.toml
resolving Python distribution...
downloading https://github.com/indygreg/python-build-standalone/releases/download/{...}/cpython-{...}.tar.zst
...
error[E0463]: can't find crate for `std`
|
= note: the `x86_64-pc-windows-msvc` target may not be installed
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.
error: Could not compile `rand_core`.
error: cargo build failed
I ran 'cargo install pyoxidizer' and got
cargo install pyoxidizer
Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to parse manifest at `%USERPROFILE%\.cargo\registry\src\github.com-1ecc6299db9ec823\pyoxidizer-0.1.1\Cargo.toml`
Caused by:
editions are unstable
Caused by:
feature `edition` is required
this Cargo does not support nightly features, but if you switch to nightly channel you can add `cargo-features = ["edition"]` to enable this feature
Edit: Would be nice if someone could also explain how this is a manifest parsing error?
Sounds like you might be using an old version of the Rust toolchain, from before the edition feature was in the stable channel.
Try this:
Try this:
rustup update
cargo install pyoxidizer
Huh I see. There's no rustup anywhere to be found. Rust just from MSYS2 in Oct. 2018, just 8 months ago. Updated it now and now it's installing fine, thanks. Here's what I had:
$ pacboy -Qi rust:x
Name : mingw-w64-x86_64-rust
Version : 1.29.2-1
Build Date : Sun, Oct 28, 2018 9:43:06 PM
...
Yeah I realized after commenting that that you might not have installed it via rustup.rs. Glad to hear that you were able to get a new enough version after updating.
8 months might not seem like a long time but quite a few things have happened in that time, including some features having become stabilized. https://github.com/rust-lang/rust/blob/master/RELEASES.md
1.35.0 is the current stable Rust toolchain indeed.
8 months might not seem like a long time but quite a few things have happened in that time, including some features having become stabilized. https://github.com/rust-lang/rust/blob/master/RELEASES.md
1.35.0 is the current stable Rust toolchain indeed.
Yup, updated my post. Would you know how to prevent Rust from embedding absolute file paths into the binary?
I'd be interested to learn what the motivation to create this was, when PyInstaller [1] already solves this (hard) problem in a fair way.
1: https://www.pyinstaller.org
1: https://www.pyinstaller.org
I tried pyinstaller, and never managed to make it work for my projects. Some libs are dynamically linked so it may cause problems.
Nuitka worked immediatly, but still dynamamically links libc and doesn't allow this nice rust/python interraction nor the possibility of future cross compilation.
Nuitka worked immediatly, but still dynamamically links libc and doesn't allow this nice rust/python interraction nor the possibility of future cross compilation.
It probably requires unpacking files onto the disk every single time? At least that's what I remember of every tool I checked out a while ago.
I think PyInstaller does, yes, if you use single file mode. The default mode simply uses an exploded directory. Not pretty, but really not a problem in practice.
How is this not a problem in practice? So much write I/O into a temp directory is slow and error-prone, and also you can't guarantee cleanup in case the process quits abnormally. And it's not like the average user will be fond of downloading and unzipping things manually. I know I absolutely loathe programs that self-extract files on a regular basis, and judging from the pains people went through to avoid it here I don't think I'm alone.
I meant the exploded directory is not a problem. And the answer to unzipping is an installer.
Cool project!
The parts I don't like:
+ Static linking
+ No extension (shared library) support
+ The module importers look over-engineered / too complicated
+ Custom Python distributions
+ Not sure what Rust gets you here, besides additional complexity
+ No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios)
Many years ago, I solved similar problems (Windows-only at the time) with an approach that:
+ Requires no static linking, absolutely no compilation of Python
+ Supports packaging any pre-installed stock Python 2 distribution, no modifications except to fix bugs in the standard library
+ Uses in-memory loading for _everything_ including Python extensions (shared libraries)
+ Is based on top of a minimal C-bootstrapping layer that can be compiled once and reused
+ Includes a better loader than the one in py2exe :-]
+ Delivers an injectable DLL or executable with absolutely no dependencies except OS built-ins
+ Works on every version of Windows from XP SP0 and beyond
The product is commercial (platform for modelling APT attacks and performing automated exfiltration pentests) but I did a presentation [1] which lays out the entire loader and the benefits of my approach in a way that is easy to independently re-engineer. I can see some benefits of static linking on Unix (even though I'd still avoid it there), but I'm convinced that it's absolutely the wrong thing to do for Windows.
[1] https://downloads.immunityinc.com/infiltrate-archives/python...
The parts I don't like:
+ Static linking
+ No extension (shared library) support
+ The module importers look over-engineered / too complicated
+ Custom Python distributions
+ Not sure what Rust gets you here, besides additional complexity
+ No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios)
Many years ago, I solved similar problems (Windows-only at the time) with an approach that:
+ Requires no static linking, absolutely no compilation of Python
+ Supports packaging any pre-installed stock Python 2 distribution, no modifications except to fix bugs in the standard library
+ Uses in-memory loading for _everything_ including Python extensions (shared libraries)
+ Is based on top of a minimal C-bootstrapping layer that can be compiled once and reused
+ Includes a better loader than the one in py2exe :-]
+ Delivers an injectable DLL or executable with absolutely no dependencies except OS built-ins
+ Works on every version of Windows from XP SP0 and beyond
The product is commercial (platform for modelling APT attacks and performing automated exfiltration pentests) but I did a presentation [1] which lays out the entire loader and the benefits of my approach in a way that is easy to independently re-engineer. I can see some benefits of static linking on Unix (even though I'd still avoid it there), but I'm convinced that it's absolutely the wrong thing to do for Windows.
[1] https://downloads.immunityinc.com/infiltrate-archives/python...
Why is "Static linking" bad? A majority of software engineers are shifting to statically linked to reduce runtime complexity.
Python 2.7 is also deprecated. New projects shouldn't really be supporting it going forward in my opinion. We're coming up on the 2020 complete end of life cycle.
Python 2.7 is also deprecated. New projects shouldn't really be supporting it going forward in my opinion. We're coming up on the 2020 complete end of life cycle.
It depends on the problem. Someone could say that static linking optimizes for runtime costs but for the problems that OP (and me) are trying to solve, I find that it increases runtime complexity since:
+ It tightly couples independent subsystems together, you can no longer see them (e.g. update) in isolation. Some of those subsystems explode when statically linked due to second-order effects (e.g. memory management).
+ Requires compilation.
+ Licensing woes that do not apply with dynamic linking.
+ Most important: Static linking reduces your runtime flexibility by being the opposite of late-binding and forces you to commit upfront. Many problems can be elegantly solved when one can take additional context into account (e.g. delay solving them until the 'how' becomes clear).
On top of that, static linking does not optimize for deployment costs, which can be substantial depending on the application and frequency of component changes.
For all these reasons, it should be avoided when possible. The counterpoint is that in practice, it takes a lot more effort to design a system that does not deploy static linking in order to solve the problems described by OP and me. Static linking is thus an easy and quick solution but it comes with many strings attached.
+ It tightly couples independent subsystems together, you can no longer see them (e.g. update) in isolation. Some of those subsystems explode when statically linked due to second-order effects (e.g. memory management).
+ Requires compilation.
+ Licensing woes that do not apply with dynamic linking.
+ Most important: Static linking reduces your runtime flexibility by being the opposite of late-binding and forces you to commit upfront. Many problems can be elegantly solved when one can take additional context into account (e.g. delay solving them until the 'how' becomes clear).
On top of that, static linking does not optimize for deployment costs, which can be substantial depending on the application and frequency of component changes.
For all these reasons, it should be avoided when possible. The counterpoint is that in practice, it takes a lot more effort to design a system that does not deploy static linking in order to solve the problems described by OP and me. Static linking is thus an easy and quick solution but it comes with many strings attached.
> The parts I don't like:
> + Static linking
It's actually what I like in this project. Dynamic linking causes many issues, especially on Linux, where you never have quite the same libs depending of the distro. If I want a standalone exe, I want static linking. If I want dynamic linking, I'll make a deb or a rpm.
> + No extension (shared library) support
> + The module importers look over-engineered / too complicated
Man, the project just came out, give it time. You're very harsh on this.
> + Custom Python distributions
Well, yes. That's why it works. Otherwise you are back to require a C compiler on the machine or use dynamic linking.
> + Not sure what Rust gets you here, besides additional complexity
- link to the custom python distrib, allowing you to use the nice rust toolchain instead of the mess that is the c toolchain or dynamic linking
- free rust / python interrop. This is very interesting.
- potentially cross compilation and even make the tools stand alone itself
> + No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios)
Python 2.7 is reaching EOL in a few months. It's perfectly reasonable to not support it. No poney.
> + Static linking
It's actually what I like in this project. Dynamic linking causes many issues, especially on Linux, where you never have quite the same libs depending of the distro. If I want a standalone exe, I want static linking. If I want dynamic linking, I'll make a deb or a rpm.
> + No extension (shared library) support
> + The module importers look over-engineered / too complicated
Man, the project just came out, give it time. You're very harsh on this.
> + Custom Python distributions
Well, yes. That's why it works. Otherwise you are back to require a C compiler on the machine or use dynamic linking.
> + Not sure what Rust gets you here, besides additional complexity
- link to the custom python distrib, allowing you to use the nice rust toolchain instead of the mess that is the c toolchain or dynamic linking
- free rust / python interrop. This is very interesting.
- potentially cross compilation and even make the tools stand alone itself
> + No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios)
Python 2.7 is reaching EOL in a few months. It's perfectly reasonable to not support it. No poney.
It's been a while since I looked at this (or at Windows, for that matter), but back in the day one thing that consistently failed for these approaches were thread initialization - after you load a DLL, every new thread gets a call to DllMain with reason DLL_THREAD_ATTACH; You can hook CreateThread and friends, but some threads are not created through those but rather through native APIs.
I don't remember the details, but ... I do remember failing to find a working solution (myself or in any of the projects I checked at the time).
I don't remember the details, but ... I do remember failing to find a working solution (myself or in any of the projects I checked at the time).
This tells how to stop it from happening.
But my problem was that I needed it to happen (e.g. if using msvcrt) and having no way to do so for DLLs “manually” loaded from memory.
But my problem was that I needed it to happen (e.g. if using msvcrt) and having no way to do so for DLLs “manually” loaded from memory.
One way is to use the Windows library loader and simply trick it to load a DLL from memory rather than disk.
If you use a method that completely bypasses the OS loader, then you have to do a lot more work to have your DLL behave like a DLL loaded by the OS would.
If you use a method that completely bypasses the OS loader, then you have to do a lot more work to have your DLL behave like a DLL loaded by the OS would.
How do you trick the loader to load from memory?
There are many ways but the one I chose is in my presentation that I linked above. Roughly, first you get an idea of the system calls involved when a DLL is loaded. Then you hook the system calls that are responsible for mapping/reading in order to redirect them to a memory buffer. Conceptually, it's straightforward. The devil is in the implementation details as they say.
I see. Thanks.
I was under the impression you needed to have some non-trivial tokens to patch NtDll in a way that kernel32 would use (which would make it unusable as a general technique), but it's been ages and I'm probably misremembering.
I was under the impression you needed to have some non-trivial tokens to patch NtDll in a way that kernel32 would use (which would make it unusable as a general technique), but it's been ages and I'm probably misremembering.
Wow, very cool. This sounds like a lot of existing projects but done in a much cleaner fashion.
If it was mentioned, I missed it, but I’m really curious to hear how things like cffi, etc. end up working. It may, with work, be possible to statically link CPython extensions, but cffi seems like a whole new packaging problem.
To avoid the X-Y problem, I’ll state a high level goal. Let’s say I want to release an application using PyQt. Can PyOxidizer handle this in any way today? In the future, does PyOxidizer plan on handling any more of the problem? Since existing Python packaging infrastructure doesn’t attempt to answer problems like this, it’d be awesome to see new initiatives that do.
If it was mentioned, I missed it, but I’m really curious to hear how things like cffi, etc. end up working. It may, with work, be possible to statically link CPython extensions, but cffi seems like a whole new packaging problem.
To avoid the X-Y problem, I’ll state a high level goal. Let’s say I want to release an application using PyQt. Can PyOxidizer handle this in any way today? In the future, does PyOxidizer plan on handling any more of the problem? Since existing Python packaging infrastructure doesn’t attempt to answer problems like this, it’d be awesome to see new initiatives that do.
Re deploying PyQt apps: check out my https://build-system.fman.io. It solves the bejeezus out of this problem.
Ah, that I have seen, thanks. I suppose that would be the end to end solution for this specific problem.
Re existing projects, there is a nice and exhaustive comparisons to other tools in the docs: https://pyoxidizer.readthedocs.io/en/latest/comparisons.html
Very strange that the pyoxidizer comparison page does not mention embed mode with Cython,
https://github.com/cython/cython/wiki/EmbeddingCython
This can be used with statically or dynamically linking Python easily.
https://github.com/cython/cython/wiki/EmbeddingCython
This can be used with statically or dynamically linking Python easily.
"""
Rust is the first programming language I've used where I feel like the language itself, the compiler, the tools around it (cargo, rustfmt, clippy, rustup, etc), and the community surrounding it all actually care about and assist me with writing high quality software. Nothing else I've used comes even close.
"""
I've been hearing nice words about Rust for a while now but it doesn't seem to catch up in popularity which I find weird.
I've been hearing nice words about Rust for a while now but it doesn't seem to catch up in popularity which I find weird.
If I ever had a problem that required system programming/low-level programming, I'd definitely try out Rust before using something like C or C++. I just haven't personally had such a problem before. Everything I've wanted to accomplish can be done in higher-level languages.
> it doesn't seem to catch up in popularity which I find weird.
What's your metric to say that ? After all, it's a system language, not a web one or scripting one, so it's not likely to go to the top trending stats very fast. Not to mention we had a stable version for only 4 years.
What's your metric to say that ? After all, it's a system language, not a web one or scripting one, so it's not likely to go to the top trending stats very fast. Not to mention we had a stable version for only 4 years.
https://www.tiobe.com/tiobe-index/
currently Rust is 38th
Seems fair to me, giving the nature of the language, the inertia of c/c++ and low level stuff, the learning curve of this domain, the popularity of web/scripting and the 4 years I mentioned.
Is PyOxidizer going to make it any harder to 'decompile' one of these Python executables? Of course, once you have the bytecode it's easy to recover the source code, but I wonder if something could be built into this tool to make it ever-so-slightly more difficult.
Nuitka has the potential to solve this issue, but it seems not ready yet.
Anyone know if/how Dropbox does it?
Nuitka has the potential to solve this issue, but it seems not ready yet.
Anyone know if/how Dropbox does it?
Dropbox uses a modified python interpreter
https://github.com/anvilventures/lookinsidethebox
https://anvilventures.com/blog/looking-inside-the-box.html
https://github.com/anvilventures/lookinsidethebox
https://anvilventures.com/blog/looking-inside-the-box.html
Nuitka can do that. Since it converts python to c, then compile the c code, you can just get the c, and compile it the way you want.
PyOxider packages and accesses the python files I think, so they are here and available.
PyOxider packages and accesses the python files I think, so they are here and available.
Just run the executable it produces through your favourite executable pack/obfuscate program. Either way it's only a delaying tactic.
Can this crossbuild? I cannot produce windows exes with pyinstaller on my linux box; can i with pyoxidizer?
I've built windows exe files using pyinstaller on Linux using docker and wine. Someone else has helped automate this. Check out https://github.com/cdrx/docker-pyinstaller
Oooh, this seems like exactly what I need. I hope it gains some traction. Trying it out right now for small util my wife needs that has to run on a machine she doesn't fully control.
Was liking the project until I got to the gotcha: the incongruous decision of using rust (or any other programming language, for that matter).
Not only that will stifle any kind of community forming around the project (not impossible, but more difficult), but also because after that point the article became yet another mostly inaccurate proselytizing of "[x] language I use because [I like it, but I'll pretend there are sound technical reasons why it's better, even though every single one is contested]".
And I ended up closing the tab entirely after this sentence "in the same way that python is technically a C application", which pretty much encapsulates my previous paragraph.
I hope I'm wrong and the project takes off; python does need better packaging and distribution. But in the meantime, I'll keep looking into python-based solutions to the problem.
Not only that will stifle any kind of community forming around the project (not impossible, but more difficult), but also because after that point the article became yet another mostly inaccurate proselytizing of "[x] language I use because [I like it, but I'll pretend there are sound technical reasons why it's better, even though every single one is contested]".
And I ended up closing the tab entirely after this sentence "in the same way that python is technically a C application", which pretty much encapsulates my previous paragraph.
I hope I'm wrong and the project takes off; python does need better packaging and distribution. But in the meantime, I'll keep looking into python-based solutions to the problem.
So, to be clear, you don't like it when projects choose to use their preferred language, and so you will search for a project which uses your preferred language?
If I'm solving a problem with Go, I will first look into solutions written in Go. If I'm solving a problem with Javascript, I'll first look into solutions written in JavaScript. If I'm solving a problem in Python, I'll first look into solutions written in Python. Etc, etc, etc.
If this project does prove to be better than something like PyInstaller, I will welcome it; I wouldn't go against the grain and would rather use a standard, de facto or official. In the meantime, I'd rather not leave an ecosystem just so I can develop for that ecosystem.
If this project does prove to be better than something like PyInstaller, I will welcome it; I wouldn't go against the grain and would rather use a standard, de facto or official. In the meantime, I'd rather not leave an ecosystem just so I can develop for that ecosystem.
I love this project, it is much needed, and your documentation/writeups are outstanding. Thank you for your work.
Very cool project! As a python developer, I completely share the concerns mentioned in this article.
From my point of view, building a new project from scratch requires a certain "marketing approach". The idea of having "The PyOxidizer's functionality rolling up into official packaging tools like pip" seems to be key to massive adoption.
Also see Facebook’s XAR: https://github.com/facebookincubator/xar
XAR is only for "Linux or macOS", with FUSE installed.
PyOxidizer offers a solution that:
- works on Windows
- can produce executable for Windows, Mac and Linux, but also potentially cross compile from any of those in the future
- results in a stand alone executable, not some filesystem abstraction
- as such, the target machine doesn't need to have anything installed
- it also offers to embed Python into Rust projects, and Rust into Python projects
A better alternative would be the excellent nuitka (nuitka.net):
- it compiles Python to C
- it's very compatible, very reliable (I battle tested it)
- works better than the alternatives like py2exe, cx_freeze, etc
- does provide a standalone executable
- more mature that both of these projects
But:
- no cross compilation
- no rust / python interoperability
- more complicated to get working on windows
So I'm really excited for PyOxidizer. It may make the task of providing a stand alone executable even easier than with nuitka, and as a bonus, may build a bridge between the Python and Rust community, which really are meant to play together.
PyOxidizer offers a solution that:
- works on Windows
- can produce executable for Windows, Mac and Linux, but also potentially cross compile from any of those in the future
- results in a stand alone executable, not some filesystem abstraction
- as such, the target machine doesn't need to have anything installed
- it also offers to embed Python into Rust projects, and Rust into Python projects
A better alternative would be the excellent nuitka (nuitka.net):
- it compiles Python to C
- it's very compatible, very reliable (I battle tested it)
- works better than the alternatives like py2exe, cx_freeze, etc
- does provide a standalone executable
- more mature that both of these projects
But:
- no cross compilation
- no rust / python interoperability
- more complicated to get working on windows
So I'm really excited for PyOxidizer. It may make the task of providing a stand alone executable even easier than with nuitka, and as a bonus, may build a bridge between the Python and Rust community, which really are meant to play together.
>cross compile from any of those in the future
that would be very helpful, I use Linux and to be able to provide executables for Windows/macOS would be nice
that would be very helpful, I use Linux and to be able to provide executables for Windows/macOS would be nice
that, and others, are mentioned on this projects comparison page https://pyoxidizer.readthedocs.io/en/latest/comparisons.html
A desktop computer in 2025 is going to have 47 redundant installations of Chromium for various electron apps, 78 separate-but-basically-identical Python runtimes as every small utility packages the entire Python interpreter + stdlib, 200,000 redundant npm packages...