PyPy has been working for me for several years now(utcc.utoronto.ca)
utcc.utoronto.ca
PyPy has been working for me for several years now
https://utcc.utoronto.ca/~cks/space/blog/python/PyPyQuietlyWorking
105 comments
I'm not sure which came first, but whichever came second is the worst naming of anything ever.
PyPy was first. It was indeed easier in the era when the Cheese Shop was called the Cheese Shop, and not PyPi.
[edit]: After some searching, I notice that PEP 301 is actually dated Oct 2002, so perhaps I'm wrong. The public/advertised name was Cheese Shop for quite some time after that, but apparently the name PyPi was at least suggested much earlier than I thought.
https://peps.python.org/pep-0301/
[edit]: After some searching, I notice that PEP 301 is actually dated Oct 2002, so perhaps I'm wrong. The public/advertised name was Cheese Shop for quite some time after that, but apparently the name PyPi was at least suggested much earlier than I thought.
https://peps.python.org/pep-0301/
For others that do not know:
This is why python has wheels.
This is why python has wheels.
Can someone explain what a "wheel" has anything to do with "cheese"? Every time I come across the term "wheel" in Python, it makes no sense to me. I believe it has something to do with Monty Python. I think I have watched the skit that involves the cheese, but it's beyond my understanding how it's related to wheel. Maybe I watched the wrong skit. Maybe it's because I don't understand Monty Python.
Cheese is traditionally made I large round blocks called “wheels”. I.E. A wheel of cheese.
So wheel is to cheese, as loaf is to bread.
https://www.chucklingcheese.co.uk/news/cheese-gift-guides/wh...
So wheel is to cheese, as loaf is to bread.
https://www.chucklingcheese.co.uk/news/cheese-gift-guides/wh...
It’s not a Monty Python thing. Some cheese is made in a wheel shape as part of the manufacturing process. If you buy some kinds of bulk cheese, you will get it in wheels. https://dibruno.com/whole-wheel-parmigiano-reggiano/
Some think it is. Excerpt from this talk by Dustin Ingram:
https://dustingram.com/talks/2018/10/23/inside-the-cheesesho... (halfway through)
PyPI is sometimes called the Cheeseshop.
This is a reference to this Monty Python skit, where a man goes into a cheeseshop which has no cheese for sale.
The joke is that when PyPI was first created, there was nothing in it. Hence, "The Cheeseshop".
But it does have stuff in it now, so it's not really even appropriate to call it that anymore.
https://dustingram.com/talks/2018/10/23/inside-the-cheesesho... (halfway through)
PyPI is sometimes called the Cheeseshop.
This is a reference to this Monty Python skit, where a man goes into a cheeseshop which has no cheese for sale.
The joke is that when PyPI was first created, there was nothing in it. Hence, "The Cheeseshop".
But it does have stuff in it now, so it's not really even appropriate to call it that anymore.
Good lord, I have never heard of the term "wheel of cheese", thanks! Probably because I rarely eat cheese. 20+ years of using Python on and off, and I finally understand what a "wheel" is.
Addendum: Every time I tried to understand Python packaging, I would come across the term "wheel" and my eyes would glaze over because I couldn't figure out what it was from context ("what the heck is a wheel?"). Maybe now I'll go back and finally understand Python packaging.
Addendum: Every time I tried to understand Python packaging, I would come across the term "wheel" and my eyes would glaze over because I couldn't figure out what it was from context ("what the heck is a wheel?"). Maybe now I'll go back and finally understand Python packaging.
While I find the tongue in cheek, punny naming in the Python ecosystem (mostly the modules these days,) charming, I find the straightforward and predictable naming in other ecosystems useful. When I'm programming, I really prefer useful. Naming hints reduce cognitive load and reduce time looking things up. When you've got arbitrary names for a bunch of things, that requires a lot more rote memorization which isn't one of my strengths.
People having fun naming their projects with references or puns or whatever might make them more likely to make the project to begin with, which is probably more important. But even within Python, it's much easier for me to remember "ok, I need install numpy and requests," than "ok I need Beautiful Soup and Twist" or whatever.
People having fun naming their projects with references or puns or whatever might make them more likely to make the project to begin with, which is probably more important. But even within Python, it's much easier for me to remember "ok, I need install numpy and requests," than "ok I need Beautiful Soup and Twist" or whatever.
It’s not that you haven’t seen enough Monty Python, it’s that you seemingly haven’t been to a supermarket before.
Ha, no, it's because I don't (or rarely) eat cheese.
Cheese was, and to a degree still is, stored and distributed in wheels. As a retail buyer you rarely encounter an entire wheel, but it's definitely a common sight in wholesale.
Because of a "wheel of cheese," usually referring to Parmezan cheese.
No, not just Parmezan. Gouda and similar cheeses often come in wheels.
as an european i can't think of any cheese that is not produced in wheels
cheese are made in wheels.
that's 1 unit of a production of cheese is called a wheel.
that's 1 unit of a production of cheese is called a wheel.
[deleted]
Then where did eggs come from?
Pythons lay eggs.
I wish I was good at coming up with Python tools… we need to complete the set, PiPy and PiPi.
Someone has beaten you to both:
- https://pypi.org/search/?q=PiPy
- https://pypi.org/project/pipi/
- https://pypi.org/search/?q=PiPy
- https://pypi.org/project/pipi/
Maybe I’ve been beaten, or maybe we’re all on the same side…
Right now my brain is scrambled...
Yeah, I had to reread the opening paragraph before I got it
I've used pypy on large codebases for years. Generally it's fine so long as you don't need any of the packages that are thin wrappers around C/fortran. It seems a lot of maintainers these days are pretty good about considering it as a target
On the other hand the memory footprint can be painful - not just the deferred garbage collection of things like weakrefs and closed files, but even regular objects. A while back I had hope that the faster cpython project would somewhat remove our need to use it, just so we could have a lower memory footprint, but that seems to have stalled
On the other hand the memory footprint can be painful - not just the deferred garbage collection of things like weakrefs and closed files, but even regular objects. A while back I had hope that the faster cpython project would somewhat remove our need to use it, just so we could have a lower memory footprint, but that seems to have stalled
> Generally it's fine so long as you don't need any of the packages that are thin wrappers around C/fortran
This is precisely where my attempt to use Pypy failed.
This is precisely where my attempt to use Pypy failed.
Why? What specifically caused you not to seek out alternatives? PyPy and libraries with CFFI seem to work okay together, but it can run python native code nearly as fast as C, so I always look for a pure python solution rather than depend on a C library.
It was a known but not fixed issue with psycopg IIRC
Fortran! Would be interested in your story on how you’re integrating Python and Fortran. That sounds very interesting
They likely mean some of the numerical packages that call out to BLAS/LAPACK or whatever as all that was originally written in Fortran. I think a lot has been converted to C now.
SciPy is a popular Python package which exposes functionality written in Fortran to Python, for example.
numpy / scipy both use fortran. Scipy has fortran as a build dependency (ie a large portion of it depends on fortran), and numpy can be built without fortran, but has large portions of optimizations in fortran.
anyone who's ever used numpy/scipy has been using python integrated with fortran.
anyone who's ever used numpy/scipy has been using python integrated with fortran.
Scipy maintainer here. Numpy has no fortran code on itself and uses lapack lite written in C. Scipy has 10% of its codebase with very old F77 libraries that we are removing slowly. Nothing against fortran but this is slowly turning into a urban legend.
The main reason fortran is still used around is fundamentally LAPACK. BLAS has long been converted to C/Assembly, openblas mkl or others. But Lapack is a lot of code to translate not because fortran is better but the original authors wrote it in F77
The main reason fortran is still used around is fundamentally LAPACK. BLAS has long been converted to C/Assembly, openblas mkl or others. But Lapack is a lot of code to translate not because fortran is better but the original authors wrote it in F77
I just recently made some Python wrappers for a Modern Fortran optimization library for derivative free optimization. https://github.com/libprima/prima
It basically had to go through Fortran -> C -> C++ (with pybind11) -> Python. At one point we had a slightly simpler setup with Fortran -> C -> Python (with ctypes), but ended up going with pybind11 after some discussion with the SciPy folks who steered us away from ctypes.
It basically had to go through Fortran -> C -> C++ (with pybind11) -> Python. At one point we had a slightly simpler setup with Fortran -> C -> Python (with ctypes), but ended up going with pybind11 after some discussion with the SciPy folks who steered us away from ctypes.
Not the OP, but I have written a fair few python wrappers around old Fortran code. There is a pretty great tool called f2py which automates almost all of the hard work. If the Fortran code isn't to weird it can completely autogenerate all the code you need to call your fortran function from python.
Fortran is actually pretty okay if you're doing matrix or vector operations! Builtin cross products, you choose how the array is indexed when you create it, can enforce functional purity to help you parallelize, etc
It is hot garbage for almost anything that isn't math though - which is okay, because it's been focused math from the start - strings and parsing an unstructured input is an exercise in pain for example. And the ecosystem is heavily geared toward math and scientific computing, so you will find yourself rolling your own stuff quite often if you deviate from that niche
It is hot garbage for almost anything that isn't math though - which is okay, because it's been focused math from the start - strings and parsing an unstructured input is an exercise in pain for example. And the ecosystem is heavily geared toward math and scientific computing, so you will find yourself rolling your own stuff quite often if you deviate from that niche
> you choose how the array is indexed when you create it,
How?
How?
https://fortran-lang.org/learn/quickstart/arrays_strings/
during array creation you can specify custom lower bound.
during array creation you can specify custom lower bound.
It is a bit sad that PyPy great achievements are mostly ignored, and it took Microsoft and Facebook stepping in, Julia and Mojo arriving into the scene, AI GPU frameworks JITs, for finally CPython to take having a JIT seriously.
I never understood why it didn't completely take over. It worked flawlessly for every use case I tried.
It always seemed to me like it a "free" way to make it faster. Why would it matter whether it was endorsed by the BDFL or PSF?
It always seemed to me like it a "free" way to make it faster. Why would it matter whether it was endorsed by the BDFL or PSF?
It doesn’t work with psycopg or many other libraries that use c extensions. That’s the main reason.
Only because they were compiled against cpython and not pypy right? If pypy had taken off/over, wouldn't it have been the other way around, where you need to do a special compile any time you want to use a C project with cpython rather than using the prebuilt pypy-compatible version?
Edit: things seem to have changed since I last looked into compiling a dependency for pypy. https://doc.pypy.org/en/latest/faq.html#do-c-extension-modul... now says they support c-extension modules without modifications
The package you mentioned, some postgres thing apparently, mentions they support pypy 3.9 and 3.10: https://www.psycopg.org/psycopg3/docs/basic/install.html
Edit: things seem to have changed since I last looked into compiling a dependency for pypy. https://doc.pypy.org/en/latest/faq.html#do-c-extension-modul... now says they support c-extension modules without modifications
The package you mentioned, some postgres thing apparently, mentions they support pypy 3.9 and 3.10: https://www.psycopg.org/psycopg3/docs/basic/install.html
By “some postgres thing”, you mean…pretty much the only way that one would connect to Postgres in Python!
There's also https://github.com/MagicStack/asyncpg
Right, I just wasn't familiar with it (never heard this name before) and the only association I could think of for this abbreviated name was psyops and cops. Turns out it's actually for postgres which, of course, is a name I very well know. Didn't realize it was a well-known library!
Oh, I'd thought some significant effort had gone into emulating the CPython C extension interface.
But maybe even with that, it's unable to support some extensions?
But maybe even with that, it's unable to support some extensions?
If you want your code to run faster, you should probably just use PyPy.
At PyCon 2015:
* https://www.youtube.com/watch?v=2wDvzy6Hgxg&t=16m52s
* https://www.youtube.com/watch?v=2wDvzy6Hgxg&t=16m52s
If someone suggested switching to a new C++ compiler with 99% of the features of the C++ standard with the selling point that it ran template metaprograms 100x faster, could you convince your company to switch? In the setting where python shines and performance matters, python is serving as the metalanguage executed at compile time for a strictly typed compiled language such as SQL, torch graph traces, jax, or z3
that only wins when it will lower costs significantly (percentage-wise) at my company :) . We don't care how much more green it would be, how fast it is, our main thing is always "how much does it cost" with everything converted to $$ (dev time, server cost, etc) and it can't change uptime, we would lose customers quickly without uptime, we have a fickle bunch.
pypy has never kept up with cPython features and releases. it was always well behind the curve and did not provide essential behaviors in a timely fashion. plus it can't run any C extensions or cython code (or has to run them with emulations that erase the speed benefits), which takes a big chunk out of the available ecosystem. Example: the situation with numpy: https://doc.pypy.org/en/latest/faq.html#what-about-numpy-num...
It keeps up pretty well, I think. But given what's in a version of Python is "what's implemented in CPython at that version", I don't see how it's possible to keep up very easily at all.
this is nonsense, cPython's development is very public and works on a multi-year schedule. It's not like Python 3.5.0 is released and that's when everyone knows what features it will have, each major version goes through at least a year of alphas and betas. Pypy spends a few years behind that and on top of it sometimes introduces new bugs that have to be reported and fixed. this is not anyone's fault, it's just for what pypy is doing to be successful, they would need a much larger maintenance ecosystem which they just dont have. The contributor ecosystem for cpython is literally ten times larger than that of pypy.
I had always assumed it was because they had exposed far too many internals over the years, making a JIT that was backwards compatible with the majority of existing code, an extremely difficult undertaking.
I assumed it was because they explicitly did not have performance as a goal, probably because performance is so bad they wanted to have a good answer and "performance is a non-goal" is the best they could do. Psychologically that is much easier to digest than "yes it's awful, sorry".
I don't think backwards compatibility can really be the reason. Python does not care that much about backwards compatibility, as seen from the extensive list of features they remove in every release:
https://docs.python.org/3/whatsnew/3.12.html#removed
I don't think backwards compatibility can really be the reason. Python does not care that much about backwards compatibility, as seen from the extensive list of features they remove in every release:
https://docs.python.org/3/whatsnew/3.12.html#removed
I’m confused now. What’s the point of PyPy if performance isn’t a goal?
It’s a completely unrelated project
Oh they’re referencing the fork and not base PyPy, which is still about performance?
The CPython project didn’t have performance as a goal.
PyPy is a different, non-PSF implementation of Python
PyPy is a different, non-PSF implementation of Python
Oh okay. I completely misunderstood the comment. Yes, CPython isn’t designed specifically for performance. I love how legible the code is.
The fact that PyPy has a JIT is kind of a side-effect. PyPy's goal is not to make a JIT compiler for Python. To be honest, the goals are broad and ambiguous, but it's something along the lines of "refining the core Python language, a subset of Python, and then see if it can be made to run Python in various other ways".
I think, that the first goal of PyPy was to compile Python to C or something like that. JIT came later.
I think, that the first goal of PyPy was to compile Python to C or something like that. JIT came later.
PyPy was a successor to Psyco, by the same author. JIT was always a core part of the project.
You may want to read this: https://www.pypy.org/posts/2018/09/the-first-15-years-of-pyp... and further down. You'll see that JIT wasn't always a core part of the project.
You just for some reason decided to use the fact that the project was renamed at some point to fight a stupid battle on the Internetes.
You just for some reason decided to use the fact that the project was renamed at some point to fight a stupid battle on the Internetes.
“One of the not so secret goals of Armin Rigo, one of the PyPy founders, was to use PyPy together with some advanced partial evaluation magic sauce to somehow automatically generate a JIT compiler from the interpreter. The goal was something like, "you write your interpreter in RPython, add a few annotations and then we give you a JIT for free for the language that that interpreter implements."”
And?
Are you sure you understand the comment you are replying to? Your replies makes less and less sense. They just look like an attempt to pick a stupid fight.
If this needs repeating: I didn't deny that the project has a JIT compiler. I denied it was it's main goal. What you quoted confirms what I claim, but you seem to be misunderstanding this quote too?
Are you sure you understand the comment you are replying to? Your replies makes less and less sense. They just look like an attempt to pick a stupid fight.
If this needs repeating: I didn't deny that the project has a JIT compiler. I denied it was it's main goal. What you quoted confirms what I claim, but you seem to be misunderstanding this quote too?
I actually think you're the one misunderstanding the quote:
"One of the not so secret goals of Armin Rigo, one of the PyPy founders, was to use PyPy together with some advanced partial evaluation magic sauce to somehow automatically generate a JIT compiler from the interpreter."
He is saying one of the founder's original goals was to automatically generate a JIT. It may not be the "main" goal but it is certainly one of the original goals per this quote.
"One of the not so secret goals of Armin Rigo, one of the PyPy founders, was to use PyPy together with some advanced partial evaluation magic sauce to somehow automatically generate a JIT compiler from the interpreter."
He is saying one of the founder's original goals was to automatically generate a JIT. It may not be the "main" goal but it is certainly one of the original goals per this quote.
It always seemed to me like they didn't have a very good relationship personal with Guido for some reason.
Guido actually said
If you want your code to run faster, you should probably just use PyPy.I don't think these statements are in conflict.
I don't think I said the opposite.
I don't see any reason for Python to want to have JIT.
There are already languages that are more suited for that kind of model, that have a long history of incremental improvement of JIT compiler design and implementation. Why is Python trying to compete in the field where it's almost certainly guaranteed to lose, and is almost certainly guaranteed to win nothing?
Having an interpreter is great for some things. Especially, if you want to have bindings to third-party libraries, which is something Python does a lot.
----
My interpretation of the events is very different from yours. I believe that Python became swarmed by people who really, really want Java, but they feel that Java isn't cool for some superficial reason. And guided by the fear of crowd opinion, they won't just use Java and be content with it. Instead, they slowly make what used to be cool into a very bad clone of Java.
And the reason why the big names you mentioned have anything to do with JIT development is because they want to ride this wave of insecure and self-doubting developers to score popularity points, and, eventually, to control the popular technology. Esp. in the case of Microsoft, it's a multi-prong offensive. They've installed their people in Python Foundation, various SIGs related to Python. They've made CPython builds and CI run on their infrastructure, preventing it from choosing free tools for their development process... Microsoft has a bad rep with people who used to like Python, and that's why they don't rebrand it into MS-Python, but really, if they wanted to, they could certainly do it.
There are already languages that are more suited for that kind of model, that have a long history of incremental improvement of JIT compiler design and implementation. Why is Python trying to compete in the field where it's almost certainly guaranteed to lose, and is almost certainly guaranteed to win nothing?
Having an interpreter is great for some things. Especially, if you want to have bindings to third-party libraries, which is something Python does a lot.
----
My interpretation of the events is very different from yours. I believe that Python became swarmed by people who really, really want Java, but they feel that Java isn't cool for some superficial reason. And guided by the fear of crowd opinion, they won't just use Java and be content with it. Instead, they slowly make what used to be cool into a very bad clone of Java.
And the reason why the big names you mentioned have anything to do with JIT development is because they want to ride this wave of insecure and self-doubting developers to score popularity points, and, eventually, to control the popular technology. Esp. in the case of Microsoft, it's a multi-prong offensive. They've installed their people in Python Foundation, various SIGs related to Python. They've made CPython builds and CI run on their infrastructure, preventing it from choosing free tools for their development process... Microsoft has a bad rep with people who used to like Python, and that's why they don't rebrand it into MS-Python, but really, if they wanted to, they could certainly do it.
Because that is how we end up with AI tooling, and Web sites written in Python, or desktop applications in Electron.
No matter how things are supposed to be, that is not how bootcamp, self called "engineers", do their programming.
Also as proven by other ecosystems, with JIT, AOT, REPLs, in the box, having an interpreter like CPython isn't a requirement for anything.
As for your MS jab, it was called IronPython.
No matter how things are supposed to be, that is not how bootcamp, self called "engineers", do their programming.
Also as proven by other ecosystems, with JIT, AOT, REPLs, in the box, having an interpreter like CPython isn't a requirement for anything.
As for your MS jab, it was called IronPython.
The next versions of CPython are expanding in two very important areas for speed. The free-threading (NoGIL) project will allow true multithreading support and the Faster CPython is trying to speedup Python and are developing a JIT. Exciting times!
for several years I used to use pypy to get the dev speed of python on the kinda crunchy problems that you'd normally need to work in a classic compiled language for.
I remember it fondly. At the time, you could find the devs on freenode and the few times I bumped into something being slower than expected they jumped on my informal bug reports and quickly recreated it and fixed it etc. Awesome! :)
I remember it fondly. At the time, you could find the devs on freenode and the few times I bumped into something being slower than expected they jumped on my informal bug reports and quickly recreated it and fixed it etc. Awesome! :)
It took me 5 minutes to realize that this isn't PyPi the package manager but PyPy, an interpreter for python.
This was very confusing, especially because they mentionned pipx, which seems related to PyPi, so it was plausible this was about PyPi.
This was very confusing, especially because they mentionned pipx, which seems related to PyPi, so it was plausible this was about PyPi.
PyPI is not a package manager. It's the Python Package Index (hence the capital "I"). It hosts the packages but the standard manager is pip.
Yeah, it's like they're shooting themselves in both feet by just naming their project badly. What a pity.
It could have been worse. Years ago a promising audio format was competing with mp3. It was better quality at any given compression and open source. The creators named it "Ogg Vorbis". Like some kind of dungeons and dragons character or something. I haven't seen it around for a long time, I think that's because the name was so stupid.
It could have been worse. Years ago a promising audio format was competing with mp3. It was better quality at any given compression and open source. The creators named it "Ogg Vorbis". Like some kind of dungeons and dragons character or something. I haven't seen it around for a long time, I think that's because the name was so stupid.
The Vorbis codec was simply not as good[1] as AAC or Opus which have since superseded it.
1. http://opus-codec.org/comparison/
1. http://opus-codec.org/comparison/
AAC is proprietary and Opus didn't exist before 2012. Vorbis came out in 2000.
The format for audio files doesn't matter much nowadays, especially with sufficient bandwidth and most people streaming them from whatever service that uses whatever it will. In the era of Napster, those early 2000s, the formats were very front-of-mind.
The format for audio files doesn't matter much nowadays, especially with sufficient bandwidth and most people streaming them from whatever service that uses whatever it will. In the era of Napster, those early 2000s, the formats were very front-of-mind.
I think the developers at all those streaming services might disagree with you about file format not mattering.
Everyone laughs at Ogg Vorbis until the +3 great-axe starts swinging and you're taking an average of 2d12+16 damage per round.
People would just have called it "ov" and moved on. Look around the tech ecosystem, lots of thriving projects with stupid names.
> I haven't seen it around for a long time,
Spotify is using it in their standalone clients.
Spotify is using it in their standalone clients.
[deleted]
Tangentially related but pipx is a silent hero in the quest for better python environment management. Takes care of the problems with bootstrapping your environment manager of choice without installing it in the python root.
I do think that PyPy is a strangely undercelebrated heroic Python effort.
Shedskin is the completely un-celebrated heroic Python effort.
Well now don't leave us hanging...
> an experimental restricted-Python to C++ programming language compiler
> an experimental restricted-Python to C++ programming language compiler
PyPy is also (as far as I know) the only Python 2 implementation still actively supported, not including OS-specific packages.
Nuitka is actively maintained and support for 2.6 and 2.7. It is the work of a single guy, and I have never used it, so I don't know much about it.
https://nuitka.net/
https://nuitka.net/
that is awesome! When it began development I never really thought PyPy would work this well, it reminds me a bit of GraalVM in that "high-arcane" magic voodoo sense of compiler/runtime development. It took 17 years of dedicated effort to get here, but the result is really impressive.
This is why many technology improvements are only possible by visionaries that are able to dream what things should look like, Alan Kay and Jobs style.
Not by those that cannot phantom anything besides what is available today running in front of them.
Android versus Longhorn, as another example.
Not by those that cannot phantom anything besides what is available today running in front of them.
Android versus Longhorn, as another example.
In that context I wish I could see ladybird in 20 years, I have a sneaking suspicion it's another one of these cases.
>403: Request Unsuccessful
>Your request cannot be satisfied.
>You appear to be trying to break this web server. Goodbye.
Sorry, I guess...
>Your request cannot be satisfied.
>You appear to be trying to break this web server. Goodbye.
Sorry, I guess...
Ok, but what CLI tools are you using where you would care about performance between CPython and PyPy?
Year or two ago, literally any.
Now with ruff and uv, much less.
Now with ruff and uv, much less.
"Literally any" would include glances, wormhole, yt-dlp, nvitop for me. For which I don't care about performance.
[deleted]
Naming things is hard but it would be nice if people tried a bit harder to avoid clashes at least within the same ecosystem.