Python JITs are coming(lwn.net)
lwn.net
Python JITs are coming
https://lwn.net/Articles/691070/
158 comments
One thing is that the "C stuff" can actually be, like, fortran stuff, because of LAPACK and BLAS and what have you, depending on the implementation your numpy is using.
No problem - let's interpret FORTAN too. What we actually do is interpret LLVM IR, so if you can compile your language to that we can interpret it.
One of the main BLAS implementations (where lots of numerical computation spends 99% of its time) is OpenBLAs, hand-crafted assembly per CPU. Main competitor is Intel MKL, closed source.
Even if you could interpret that code, your JIT-ing is not going to create exactly the same assembly and will thus perform worse. You can very easily get a C implementation that you can interpret easily but at least as output by a C compiler, that performance is nowhere near where you want to be.
(I realize this is not a problem though, just have the JIT call the libraries, as long as the wrapper can be interpreted you are good.)
Even if you could interpret that code, your JIT-ing is not going to create exactly the same assembly and will thus perform worse. You can very easily get a C implementation that you can interpret easily but at least as output by a C compiler, that performance is nowhere near where you want to be.
(I realize this is not a problem though, just have the JIT call the libraries, as long as the wrapper can be interpreted you are good.)
This is more difficult than you might think. LLVM IR more like a family of closely related, but poorly-specified languages, than a language of its own.
There is an interpreter in the LLVM codebase, but this is essentially only possible because it is part of the codebase and so tracks the rest of the project very closely.
There is an interpreter in the LLVM codebase, but this is essentially only possible because it is part of the codebase and so tracks the rest of the project very closely.
This is more difficult than you might think.
Note that chris is not talking in the abstract, but about an existing codebase: https://github.com/graalvm/sulong
Note that chris is not talking in the abstract, but about an existing codebase: https://github.com/graalvm/sulong
OK, that's impressive.
JIT-compiling your C code covers one of the two reasons a Python programmer might write C code (dropping down to C to do some mathematical operation faster).
How do you handle the other reason people write C code-- to bridge Python code to an external native library? In that case, you'd be JIT-compiling the bridge itself, but your native library remains "opaque [insert language here] stuff" that the JIT can't do anything with.
How do you handle the other reason people write C code-- to bridge Python code to an external native library? In that case, you'd be JIT-compiling the bridge itself, but your native library remains "opaque [insert language here] stuff" that the JIT can't do anything with.
Either you can interpret the external native library as well, if you have the source code, or you can leave the calls to that library as normal native calls generated by the JIT.
So yes, that'll be an optimisation barrier, but it's the same barrier as you would already have if you were using an interpreter for Python making native calls to the same library. So it won't be any slower than our current best case.
(In fact the JIT can do some clever things to make the native call more efficient than in general, as it can do things like schedule values to be in the correct registers from the start rather than copying them in place just for the call, but that's probably not a significant source of inefficiency anyway).
So yes, that'll be an optimisation barrier, but it's the same barrier as you would already have if you were using an interpreter for Python making native calls to the same library. So it won't be any slower than our current best case.
(In fact the JIT can do some clever things to make the native call more efficient than in general, as it can do things like schedule values to be in the correct registers from the start rather than copying them in place just for the call, but that's probably not a significant source of inefficiency anyway).
What of JITing x86 assembly into x86?
Yes that's another option. We already do that for inline assembly in C programs.
This is basically how QEMU works when it's not using KVM
If you're using C for interop and not performance, it doesn't matter (much) if the JIT can optimize for it or not. You'll still be no worse off than you are today with CPython + C extensions.
[deleted]
At that point what do you gain versus porting all the C stuff to the host language so it can be JIT compiled?
I'm not sure what you mean - you gain not having to port anything, which sounds pretty great to me.
And for the Ruby C extensions we tried not only did we not have to port them from C to Ruby, they actually ran faster than either the compiled C version or the pure Ruby version because we can inline and optimise between the two languages.
And for the Ruby C extensions we tried not only did we not have to port them from C to Ruby, they actually ran faster than either the compiled C version or the pure Ruby version because we can inline and optimise between the two languages.
I mean the cost of porting vs writing a whole new C interpreter - sounds like a lot of work.
This way it's a lot of work for one small team to write the C interpreter, compared to a still quite a lot of work for every team with a C extension that would need to port it to Python.
Writing a C interpreter isn't that bad, especially considering you can co-opt Clang's frontend and use it to do all the syntax and semantic analysis.
There's a reason new languages, compilers are made rather than everyone just porting their code to the Assembler of new CPU.
Same reason applies here.
Same reason applies here.
In the case of python, we already have Cython that allows the porting of Python to C. Why do work creating the inverse (with potentially much harder semantics to JIT) when we can already go from Python to C.
Agreed! JIT's are interesting but we use Cython in production to build Userify Enterprise (SSH key management) (self-hosted distributable). It works incredibly well and we are thrilled with the result. (We do require a bunch of third party pip installs rather than wrap them into the binary.) This was partly because one thing that we had trouble with was lots of imports, especially for (ironically) compiled C libraries. It's a lot easier and probably safer to just give a user a script of installs (or a requires manifest.) Someday soon we'll probably even make it installable via pip and bring things full circle. (We're exploring a free five server version.)
Userify's architecture is probably a bit unusual. It's distributed as a single binary that uses an external redis server for locking and stores data in either S3 or a local filesystem (NFS, EBS, iSCSI, etc.) When it first launches, it automatically installs any missing python prereqs (if pip is available) and explodes out a web server static file repo in case you want to front-end it with nginx (etc) and then starts up multiple web servers (HTTPS, HTTP, alternative ports, etc). We've found Python to work extremely well for all of this. We've looked at a few other languages that we think fit our core requirements (incl lua, scala, and haskell) but Python works pretty much perfectly for this use case, we can scale it horizontally using Redis as a synchronization mechanism, and the GIL hasn't caused any problems for us because we just add more processes as needed. Cython is really fantastic and 'just works'.
Things aren't perfect.. we're still on Python 2.x, but we're going to be moving forward as soon as all of the third-party libraries we rely on move to 3.
Userify's architecture is probably a bit unusual. It's distributed as a single binary that uses an external redis server for locking and stores data in either S3 or a local filesystem (NFS, EBS, iSCSI, etc.) When it first launches, it automatically installs any missing python prereqs (if pip is available) and explodes out a web server static file repo in case you want to front-end it with nginx (etc) and then starts up multiple web servers (HTTPS, HTTP, alternative ports, etc). We've found Python to work extremely well for all of this. We've looked at a few other languages that we think fit our core requirements (incl lua, scala, and haskell) but Python works pretty much perfectly for this use case, we can scale it horizontally using Redis as a synchronization mechanism, and the GIL hasn't caused any problems for us because we just add more processes as needed. Cython is really fantastic and 'just works'.
Things aren't perfect.. we're still on Python 2.x, but we're going to be moving forward as soon as all of the third-party libraries we rely on move to 3.
Extensibility from C isn't there for people to accelerate their slow code, it's much rather there to cooperate with existing libraries, and C is the lowest common denominator there.
I think CFFI is supposed to be able to link between Pypy and C code with minimal code (same should be possible with other JIT Pythons).
Very interesting point of view.
I see a fourth option:
- compile the scripting language to C (like Crystal for Ruby)
I see a fourth option:
- compile the scripting language to C (like Crystal for Ruby)
The biggest gains of JIT is that you can use knowledge gained at runtime to generate specialized code and skip lots of the runtime checks that are typical of dynamic languages. You can't get rid of these checks if you compile the source language to C so from this point of view compiling to C is going to be just as slow as writing a traditional interpreter in C.
Yes but you can bootstrap the compiler.
The implicit runtime checks in a Python program get converted into explicit if-statements if you use an ahead-of-time compiler to compile it to a static language like C. Bootstrapping will not be able to magically get rid of these if statements.
Not to mention, bootstrapping is about the compiler executable and the problem I was talking about applies to any attempt to use an ahead-of-time compiler for a dynamic language.
Not to mention, bootstrapping is about the compiler executable and the problem I was talking about applies to any attempt to use an ahead-of-time compiler for a dynamic language.
Modern Smalltalk environments are bootstrapped as well.
Python already has this through Cython: http://cython.org/
Crystal compiles directly to native code, it doesn't rely on C compilers for the last step.
Is there anything blocking Jython from using this same approach?
Jython isn't (yet) implemented on Truffle, I believe (and would require basically a rewrite).
The approach chrisseaton describes is generally only feasible if you're implementing a Futamura projection à la PyPy or Truffle.
That said, it's only a matter of time before a high-performance Python pops up on Truffle and JITs extensions like RubyTruffle.
The approach chrisseaton describes is generally only feasible if you're implementing a Futamura projection à la PyPy or Truffle.
That said, it's only a matter of time before a high-performance Python pops up on Truffle and JITs extensions like RubyTruffle.
There is however ZipPy which is a Py3 implementation built on Truffle, though obviously not directly comparable to the Py2 Jython.
The Truffle implementation of Ruby can be plugged into a Truffle implementation of C, in order to maintain compatibility with C extensions
http://chrisseaton.com/rubytruffle/cext/
Has anyone in the Python community thought of doing something similar, e.g. using PyPy to build a C interpreter, maybe re-using front-end components (pre-processor, parser, type-checker, etc.) from an existing C compiler? In fact, it might be useful to build an interpreter for something like LLVM IR, or even x86 machine code, in order to gain access to a bunch of existing languages.
Once access is gained at that low (ABI?) level, abstractions and interfaces can be built to hide the horribleness. The performance would initially be terrible, but some targetted, profiling-guided optimisation might get it down to reasonable levels; in a similar way to a JS interpreter adding specific optimisations to make asm.js code run fast.
http://chrisseaton.com/rubytruffle/cext/
Has anyone in the Python community thought of doing something similar, e.g. using PyPy to build a C interpreter, maybe re-using front-end components (pre-processor, parser, type-checker, etc.) from an existing C compiler? In fact, it might be useful to build an interpreter for something like LLVM IR, or even x86 machine code, in order to gain access to a bunch of existing languages.
Once access is gained at that low (ABI?) level, abstractions and interfaces can be built to hide the horribleness. The performance would initially be terrible, but some targetted, profiling-guided optimisation might get it down to reasonable levels; in a similar way to a JS interpreter adding specific optimisations to make asm.js code run fast.
Something like Sulong¹ in RPython would be really cool and probably entirely feasible—I too am curious if that's on the PyPy team's radar.
¹ https://github.com/graalvm/sulong
¹ https://github.com/graalvm/sulong
Sulong is a really good idea -- I wish I'd thought of it first! We've had a student do a small project looking at something equivalent for RPython. There are, as expected, no show-stoppers yet, but I have no idea how far we'll be able to go with the limited resources we have to throw at the problem.
I must say I'm always a little annoyed by the split in Python community around the scientific stack vs the rest. There are two ecosystems, two packaging tools etc (which is fine), but the insistence that there are no other python users that are worth considering (both sides are guilty) is really frustrating.
Former Cython developer here.
To get some perspective to this, consider the alternatives for a scientific programmer: MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia -- all specifically made for scientific programming with 0% general purpose/web/etc. development going on.
So I would say that the scientific Python community has been doing extremely well in terms of even using a language that isn't designed ground up for scientific computing.
I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) -- I'll refrain, I just wanted to say that were you see something and get frustrated, I see the same picture and think it's actually an incredible success, to bring so many scientists onto at least the same ball park as other programmers, even if they are still playing their own game.
To get some perspective to this, consider the alternatives for a scientific programmer: MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia -- all specifically made for scientific programming with 0% general purpose/web/etc. development going on.
So I would say that the scientific Python community has been doing extremely well in terms of even using a language that isn't designed ground up for scientific computing.
I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) -- I'll refrain, I just wanted to say that were you see something and get frustrated, I see the same picture and think it's actually an incredible success, to bring so many scientists onto at least the same ball park as other programmers, even if they are still playing their own game.
As an aside, these sorts of comments about the needs of a "scientific programmer" always irk me.
I've been doing scientific software development now for 20 years. I do non-numerical scientific computing, originally structural biology, then bioinformatics, and now chemical informatics, the last dominated by graph theory.
I rarely use NumPy and effectively never the languages you mentioned. Last year on one project I did use a hypergeometric survival function from SciPy, then re-implemented it in Python so I wouldn't have the large dependency for what was a few tens of lines of code.
Biopython, as another example, has almost no dependencies on NumPy, and works under PyPy.
I've been doing scientific software development now for 20 years. I do non-numerical scientific computing, originally structural biology, then bioinformatics, and now chemical informatics, the last dominated by graph theory.
I rarely use NumPy and effectively never the languages you mentioned. Last year on one project I did use a hypergeometric survival function from SciPy, then re-implemented it in Python so I wouldn't have the large dependency for what was a few tens of lines of code.
Biopython, as another example, has almost no dependencies on NumPy, and works under PyPy.
It would be awesome if you have a blog post or another resource that contains some of your work experience that you can link? I'd be very interested.
The best I can offer are my previous HN comments on the topic, at https://hn.algolia.com/?query=dalke%20numpy&sort=byPopularit... and http://www.dalkescientific.com/writings/diary/archive/2011/1... .
I don't see what's awesome or all that interesting about it.
I don't see what's awesome or all that interesting about it.
> MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia
You are so correct. I have not used Julia, but what do the first four have in common? As languages, they suck. Each in their own way can be used to do amazing, incredible things. But from a development perspective, they are pure torture to code in.
The joy of writing scientific code in Python is that there is a whole set of users, the majority really, who have nothing to do with science. The language must stand on its own, so Python cannot suck.
You are so correct. I have not used Julia, but what do the first four have in common? As languages, they suck. Each in their own way can be used to do amazing, incredible things. But from a development perspective, they are pure torture to code in.
The joy of writing scientific code in Python is that there is a whole set of users, the majority really, who have nothing to do with science. The language must stand on its own, so Python cannot suck.
> I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..)
Please do! I'm a PhD student in CS, and I don't think I "get" scientific computing (I'm in compilers myself).
Please do! I'm a PhD student in CS, and I don't think I "get" scientific computing (I'm in compilers myself).
I am a PhD in CS, specifically in Programming Languages and Parallel Programming and I believe I do get scientific computing.
Most people doing it did not have a formal CS education. They are biology, physics, mathematics or chemistry majors that have had one or two courses on programming, from other scientific programmers.
There are two main families, one that comes from the Fortran background, which still writes programs like they did in the 80s, with almost no new tooling. Programs are written for some time, and then they are scheduled for clusters that spend months calculating whatever it is.
The other family of scientific programmers, which I believe is the majority, uses a tool like Matlab, or more recently R, to dynamically inspect and modify data (RStudio is a Matlab/Mathematica-like friendly environment for this task) and use libraries written by more proficient programmers to perform some kind of analysis (either machine learning, DNA segmentation, plotting or just basic statistics).
Most of these programmers know 1 or 2 languages (maybe plus python and bash for basic scripting). They write programs that are relatively small and the chances of someone else using that code is low. Thus, the deadline pressure is high and code maintainability is not a priority.
For a non-CS programmer, learning a new programming language is almost impossible, because they are used to that way of doing things, and those libraries. They take much more time to adjust to new languages because they do not see the language logically, like anyone who had a basic compiler course.
Given this context, web apps, rest APIs and all the other trending tech in IT are not commonly used in scientific programming, because they typically do not need it (when they do, they learn it). Datasets are retrieved and stored in CSV and processed in one of those environments (or even in julia or python-pandas).
Most people doing it did not have a formal CS education. They are biology, physics, mathematics or chemistry majors that have had one or two courses on programming, from other scientific programmers.
There are two main families, one that comes from the Fortran background, which still writes programs like they did in the 80s, with almost no new tooling. Programs are written for some time, and then they are scheduled for clusters that spend months calculating whatever it is.
The other family of scientific programmers, which I believe is the majority, uses a tool like Matlab, or more recently R, to dynamically inspect and modify data (RStudio is a Matlab/Mathematica-like friendly environment for this task) and use libraries written by more proficient programmers to perform some kind of analysis (either machine learning, DNA segmentation, plotting or just basic statistics).
Most of these programmers know 1 or 2 languages (maybe plus python and bash for basic scripting). They write programs that are relatively small and the chances of someone else using that code is low. Thus, the deadline pressure is high and code maintainability is not a priority.
For a non-CS programmer, learning a new programming language is almost impossible, because they are used to that way of doing things, and those libraries. They take much more time to adjust to new languages because they do not see the language logically, like anyone who had a basic compiler course.
Given this context, web apps, rest APIs and all the other trending tech in IT are not commonly used in scientific programming, because they typically do not need it (when they do, they learn it). Datasets are retrieved and stored in CSV and processed in one of those environments (or even in julia or python-pandas).
You're painting an awfully dark picture of scientists' skills. Having been on both sides, I believe the deciding factor is simply the availability of libraries.
If you're doing web development you have an insane amount of languages to chose from because after String, Array, and File are implemented, HTTP is next. Having done a bit of web development, I'd also say a typical project only uses a subset of libraries that is surprisingly small.
Scientific computing is quite different: a paper in structural biology (my former stomping grounds) can easily require a few dozen algorithms that each once filled a 10-page paper. These could easily be packaged as libraries, but it's a niche so it rarely happens. Newer language quite often don't even have a robust numeric library. Leave the beaten tracks and your workload just increased by a magnitude.
That's also why science, unlike "general purpose" programming, often uses a workflow that connects five or more languages or so: a java GUI, python for network/string/fileIO, maybe R for larger computations, all held together by a (typically too long) shell script.
But these workflows are getting better. There's a build tool that formalizes the pipeline somewhat (I forgot the name) and APIs are surprisingly common. The reason why csv will never die is that the data fetched from APIs is usually more static than it is in a typical web app (-> local cache needed) and that scientists often work with data that just isn't a good fit for a database. Postgres just doesn't offer anything that enriches a 15MB gene sequence.
If you're doing web development you have an insane amount of languages to chose from because after String, Array, and File are implemented, HTTP is next. Having done a bit of web development, I'd also say a typical project only uses a subset of libraries that is surprisingly small.
Scientific computing is quite different: a paper in structural biology (my former stomping grounds) can easily require a few dozen algorithms that each once filled a 10-page paper. These could easily be packaged as libraries, but it's a niche so it rarely happens. Newer language quite often don't even have a robust numeric library. Leave the beaten tracks and your workload just increased by a magnitude.
That's also why science, unlike "general purpose" programming, often uses a workflow that connects five or more languages or so: a java GUI, python for network/string/fileIO, maybe R for larger computations, all held together by a (typically too long) shell script.
But these workflows are getting better. There's a build tool that formalizes the pipeline somewhat (I forgot the name) and APIs are surprisingly common. The reason why csv will never die is that the data fetched from APIs is usually more static than it is in a typical web app (-> local cache needed) and that scientists often work with data that just isn't a good fit for a database. Postgres just doesn't offer anything that enriches a 15MB gene sequence.
I worked in the academia for a few years about a decade ago and nowadays interact with biology research in the industry for the last couple of years.
The way he painted the scientists skills matches my experience thus far.
The way he painted the scientists skills matches my experience thus far.
Yes, scientists programming skills (as averaged over population) suck. Factor 1: Programming not credited in itself or reviewed in publishing process. Factor 2: Often little education in or focus on programming, relative to wall clock time spent doing it.
But I don't think that is only fixed by more education and making scientists behave more like programmers. I think that to change things one also needs far better alternatives than the options available today, so that people are really encouraged to switch. Somehow, these must be written by people who know their CS and can write compilers, yet engage with the why scientific computing is a mess on the tool side too, not dismiss it as laziness.
I started out as a programmer, I have contributed to Cython, past two years have been pure web development in a startup. So I know very well why MATLAB sucks. Yet, the best tool I have found myself for doing numerical computing is a cobbled mess of Fortran, pure C, C/assembly code generated by Python/Jinja templates, Python/NumPy/Theano...
The scientific Python community and Julia community has been making great progress, but oh how far there is left to go.
But I don't think that is only fixed by more education and making scientists behave more like programmers. I think that to change things one also needs far better alternatives than the options available today, so that people are really encouraged to switch. Somehow, these must be written by people who know their CS and can write compilers, yet engage with the why scientific computing is a mess on the tool side too, not dismiss it as laziness.
I started out as a programmer, I have contributed to Cython, past two years have been pure web development in a startup. So I know very well why MATLAB sucks. Yet, the best tool I have found myself for doing numerical computing is a cobbled mess of Fortran, pure C, C/assembly code generated by Python/Jinja templates, Python/NumPy/Theano...
The scientific Python community and Julia community has been making great progress, but oh how far there is left to go.
I agree, this is also one of the things that drives me against C and more into saner programming languages.
Because the majority of programmers in areas where software isn't the core product being sold, don't spend one second thinking about code quality.
As such tooling that on one side is more forgiving while allowing for fast prototyping, but at the same time enforces some kind of guidelines is probably the way to improve the current workflows.
Because the majority of programmers in areas where software isn't the core product being sold, don't spend one second thinking about code quality.
As such tooling that on one side is more forgiving while allowing for fast prototyping, but at the same time enforces some kind of guidelines is probably the way to improve the current workflows.
[deleted]
I've spent large parts of my career floating on the edges of academia and have had to interface with code written by academics many times, and : oh jesus it's almost always a huge mess.
You want to process (typically, with simple arithmetical operations) huge arrays of numbers. Imagine that you want to sum two arrays of 800 MB of floating point numbers each. This is one step inside the loop of your algorithm.
You can do that natively in C, and the result is very fast. You can do that natively in Fortran. And in Matlab, etc.
You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower.
You can do that natively in C, and the result is very fast. You can do that natively in Fortran. And in Matlab, etc.
You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower.
> You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower.
NumPy + Cython would beg to differ.
NumPy + Cython would beg to differ.
Of course, but not natively. Try doing that in plain python using a list of 8000 lists of 8000 lists of 3 numbers (e.g. a mid-resolution microscopy image).
NumPy is about one order of magnitude slower in very many situations. Cython is in some ways a reimplementation of Fortran in places -- I mean, you can consider it a better alternative but have to use features that are not in Python.
Personally I gave up writing numerical code in Cython and instead wrote it in Fortran and merely wrapped it with Cython...
But yes, being 2x-10x slower is something one can live with for productivity, vs sometimes 1000x slower of Python.
Personally I gave up writing numerical code in Cython and instead wrote it in Fortran and merely wrapped it with Cython...
But yes, being 2x-10x slower is something one can live with for productivity, vs sometimes 1000x slower of Python.
+1, I'm a PhD student in programming languages/PL design
Julia has tons of general purpose development going, including an mvc web framework (genie.jl) and a reactive web app framework called (escher.jl)
First, sorry for being a bit heavy on the hyperbole and saying "0%"; that is almost guaranteed to be wrong as a statement and also let me say I don't know Julia very well at all.
I want to clarify that when I say "0% web development going on", I don't mean that scientific programmers don't do web development (they do! a lot!), I meant that people don't pick it those languages in general if they are only doing web development without a numerical/scientific/statistical aspect to it.
What would be interesting is, do you know of any teams or companies using Julia in anger in a non-scientific setting, with programmers from a non-scientific background?
The scientific Python community certainly makes good use of all the Python web tools!, and being a "scientific stack" in no way precludes the need for general purpose frameworks that can also be used by others. It's at least as much about people and community and habits as about the tooling...
I want to clarify that when I say "0% web development going on", I don't mean that scientific programmers don't do web development (they do! a lot!), I meant that people don't pick it those languages in general if they are only doing web development without a numerical/scientific/statistical aspect to it.
What would be interesting is, do you know of any teams or companies using Julia in anger in a non-scientific setting, with programmers from a non-scientific background?
The scientific Python community certainly makes good use of all the Python web tools!, and being a "scientific stack" in no way precludes the need for general purpose frameworks that can also be used by others. It's at least as much about people and community and habits as about the tooling...
I think its just a matter of time before this happens (its mature enough or an important app is released).
Julia is designed for general purpose computing from day one, but this community is not dealing with the same painpoints that scientists have been.
Julia is designed for general purpose computing from day one, but this community is not dealing with the same painpoints that scientists have been.
> do you know of any teams or companies using Julia in anger in a non-scientific setting, with programmers from a non-scientific background?
It's catching on among early-adopters in finance. What do you consider that?
It's catching on among early-adopters in finance. What do you consider that?
Even APL has a couple of web frameworks...
Having a few libraries of each kind (of varying quality and with very small adoption) != tons.
Having a few libraries of each kind (of varying quality and with very small adoption) != tons.
Slight aside -- I am a long-time user of Cython and would love to contribute, but poking around the source code feels daunting -- like there is a huge learning curve to be able to contribute, and that other contributors will only view newcomers as an annoyance who will drain their time with questions.
Given all this, what's the best way for someone to pick up and start contributing to Cython?
Given all this, what's the best way for someone to pick up and start contributing to Cython?
It definitely is daunting; in particular because Cython isn't so much a goal as itself as a tool that developers are motivated to work on it because they use it in some other project they care about (e.g., lxml, Sage, ...). So it's been pragmatic development.
If you don't already, subscribe to and start to follow cython-devel. First step is probably repeat the question there for more up-to-date info than what I can give (I don't even follow any longer).
Think about what you want to achieve/change in Cython. A new feature may be easier than a bugfix, though I'm not sure how many "low hanging features" are left at this point.... Anyway, make sure you understand what the change would involve in changes in the C code. Write a testcase that uses Cython the way you would want it to work (elicit the failure/bug/feature); look at the C code that Cython generates, and make sure you understand why that C code is the wrong code and that you know how you'd want it to look. (Understanding the generated C code at this level and read it almost fluently may take a little bit of getting used to but is an absolute requirement for working with Cython -- eventually you look past all the __pyx_ everywhere).
Then somehow try to beat the Cython codebase into submission in generating that C code... as you repeat this process you'll gradually learn the code.
It seems like Robert Bradshaw and Stefan Behnel are still around, they are very capable and friendly people and I learned a lot of what I know about programming from them, they were very welcoming to me as a new contributor.
If you don't already, subscribe to and start to follow cython-devel. First step is probably repeat the question there for more up-to-date info than what I can give (I don't even follow any longer).
Think about what you want to achieve/change in Cython. A new feature may be easier than a bugfix, though I'm not sure how many "low hanging features" are left at this point.... Anyway, make sure you understand what the change would involve in changes in the C code. Write a testcase that uses Cython the way you would want it to work (elicit the failure/bug/feature); look at the C code that Cython generates, and make sure you understand why that C code is the wrong code and that you know how you'd want it to look. (Understanding the generated C code at this level and read it almost fluently may take a little bit of getting used to but is an absolute requirement for working with Cython -- eventually you look past all the __pyx_ everywhere).
Then somehow try to beat the Cython codebase into submission in generating that C code... as you repeat this process you'll gradually learn the code.
It seems like Robert Bradshaw and Stefan Behnel are still around, they are very capable and friendly people and I learned a lot of what I know about programming from them, they were very welcoming to me as a new contributor.
Hi, I recognize you as a PyPy developer. Do you think it's fair to write that "no one uses PyPy"?. I feel there are quite many companies using PyPy, mostly for web backend. You don't see many usages in other usages like Linux distros or ad-hoc scripts, even when the C extensions are not used, because 1) In 95% of cases, the performance is good enough, 2) the warmup slowdown dwarfs the overall gains.
Anyway, I think PyPy doesn't get the attention it deserves - the performance gains are fantastic and the vision of the Python ecosystem as Python-only packages with occasional lightweight C libraries integrated with cffi looks very nice to me.
Anyway, I think PyPy doesn't get the attention it deserves - the performance gains are fantastic and the vision of the Python ecosystem as Python-only packages with occasional lightweight C libraries integrated with cffi looks very nice to me.
I mean, it's obviously unfair. Even in the scientific community someone uses PyPy. As far as download statistics go, I think between 0.5-1% of downloads from PyPI are for PyPy. This is far from "majority" but far from "noone" as well. For what is worth enough big players use PyPy that the consulting company that I run can stay afloat, so I somehow doubt that's the case :-)
From an outside perspective with no skin in the game (other than using Python profusely and just generally wanting it to be the "best" possible Python it can be) I wouldn't be so quick to call it obviously unfair.
The exact phrasing was:
> PyPy is ten years old at this point, but to a first approximation, no one is using it.
Using your 0.5% - 1.0% PyPI metric as an approximation for how many people are actually using pypy, I think it's reasonable to say that. I personally would definitely have phrased it differently, but if >=99% of the market isn't using PyPy, then it is far, far, far removed from the "mainstream" market. In fairness, the Python community is, on the whole, large enough to give you a reasonably sustainable niche, but it pales in comparison to the market itself.
It's a bit like comparing Facebook to CouchSurfing, to be honest. You know about it, maybe even have some friends who have done it, and hey, it's still 3 million people, but in comparison to Facebook... no one uses it. And that's not a value judgment or anything, it's just a scale of comparison kind of thing.
That being said, I share your frustration about the deep schism between web vs scientific in the Python community, and I think it's really a shame for the community as a whole. Unfortunately I'm not sure that's going to change very soon: the web side of things has a tremendous amount of momentum (and money) invested in "their" architecture, and at the same time, the scientific side of things has far, far less patience for pain points in their language tooling.
Put differently, if your background and job is programming, you're more likely to view "dealing with this programming problem" as actual work, but if your background and job is "I have this data, and I need to analyze it", then "fiddling with this programming problem" is, at best, a frustrating distraction from your actual task. And I think both arenas need to have a better appreciation for the other: as programmers, our tools generally really do suck for everyone, we're just used to it; as data scientists, we are woefully under-aware of how difficult these programming problems can be. Some more unity would be tremendously beneficial for all.
And then on top of it all, there's this whole group of weirdos using Python for stuff like desktop applications (I happen to be in this camp). Good luck finding a packaging and deployment solution there!
The exact phrasing was:
> PyPy is ten years old at this point, but to a first approximation, no one is using it.
Using your 0.5% - 1.0% PyPI metric as an approximation for how many people are actually using pypy, I think it's reasonable to say that. I personally would definitely have phrased it differently, but if >=99% of the market isn't using PyPy, then it is far, far, far removed from the "mainstream" market. In fairness, the Python community is, on the whole, large enough to give you a reasonably sustainable niche, but it pales in comparison to the market itself.
It's a bit like comparing Facebook to CouchSurfing, to be honest. You know about it, maybe even have some friends who have done it, and hey, it's still 3 million people, but in comparison to Facebook... no one uses it. And that's not a value judgment or anything, it's just a scale of comparison kind of thing.
That being said, I share your frustration about the deep schism between web vs scientific in the Python community, and I think it's really a shame for the community as a whole. Unfortunately I'm not sure that's going to change very soon: the web side of things has a tremendous amount of momentum (and money) invested in "their" architecture, and at the same time, the scientific side of things has far, far less patience for pain points in their language tooling.
Put differently, if your background and job is programming, you're more likely to view "dealing with this programming problem" as actual work, but if your background and job is "I have this data, and I need to analyze it", then "fiddling with this programming problem" is, at best, a frustrating distraction from your actual task. And I think both arenas need to have a better appreciation for the other: as programmers, our tools generally really do suck for everyone, we're just used to it; as data scientists, we are woefully under-aware of how difficult these programming problems can be. Some more unity would be tremendously beneficial for all.
And then on top of it all, there's this whole group of weirdos using Python for stuff like desktop applications (I happen to be in this camp). Good luck finding a packaging and deployment solution there!
PyPy finally got $200K of funding, from Mozilla.[1] They're going to support Python 3.5. PyPy ought to be able to use a version of NumPy written in Python. If someone can pound Python's little tin god into making Python's new type annotations have sane semantics (as in enforcing the typing), then NumPy in Python could come up to NumPy speed.
The amount of C code used with Python declined somewhat with Python 3, because the C interface changed and things had to be reimplemented. There's pymysql, for example, which is a database connector in pure Python. No more need for the C version.
[1] https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from...
The amount of C code used with Python declined somewhat with Python 3, because the C interface changed and things had to be reimplemented. There's pymysql, for example, which is a database connector in pure Python. No more need for the C version.
[1] https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from...
I wish this will happen to ruby. For me, and this is personal, ruby without jruby and truffle is dead
While I still think it a bit unfair, the exact quote is "...but to a first approximation, no one is using it". So I think (assuming fijal's numbers of 0.5-1% are roughly correct) it is not completely unreasonable to say that PyPy adoption is very limited to date, which was the point of what he was saying (although I think it could have been said more diplomatically).
Yes, some people wondered if PyPy is "the future of Python" - but it didn't take over the Python ecosystem. If the initiatives mentioned in the article will yield a 100%-CPython-compatible implementation that is always faster and doesn't have big warmup slowdown, it could be "future of Python".
I've been testing it Gtk stuff each release for a while now, it seems really close to working with the things I need (CPython libs). About 3 years ago, I couldn't have guessed it would come that far + this is as a fan/lurker of their mailing list :)
I'm also surprised that it mostly works with extensions written for CPython. Unfortunately there's a big difference between working in 99.9% of cases vs 100%...
Actual target is not 100%, but CPython interversion agreement, because CPython versions (say 3.4 and 3.5) are not 100% compatible with each other. I think PyPy can reach that target some day.
> the vision of the Python ecosystem as Python-only packages with occasional lightweight C libraries integrated with cffi looks very nice to me.
I have the opposite view. I'd rather have all of the important libraries written in a language with a safer type system than Python's. C's type system has some safety issues, but it's still safer than Python's.
If Python 3's optional type hints could actually be enforced at runtime, I'd be sold on using Python for pretty much everything.
I have the opposite view. I'd rather have all of the important libraries written in a language with a safer type system than Python's. C's type system has some safety issues, but it's still safer than Python's.
If Python 3's optional type hints could actually be enforced at runtime, I'd be sold on using Python for pretty much everything.
> C's type system has some safety issues, but it's still safer than Python's.
Uhm. An error in Python code code can lead to a runtime crash with an exception or a bad result ...most of the time. An error in C can lead to anything from an impossibly hard to diagnose memory leak to an exploitable buffer overflow... most of the time.
C may be be "statically typed", but I think the number of people capable of writing (and maintaining!) secure C code is incredibly low. Heck, even the openssl devs failed at this at least once.
And the fact that C is "static" and Python is "dynamic" does not make C's type system safe. Heck, even js's type "system" is safer than C's if when you use the word safe you refer to, you know, security.
Uhm. An error in Python code code can lead to a runtime crash with an exception or a bad result ...most of the time. An error in C can lead to anything from an impossibly hard to diagnose memory leak to an exploitable buffer overflow... most of the time.
C may be be "statically typed", but I think the number of people capable of writing (and maintaining!) secure C code is incredibly low. Heck, even the openssl devs failed at this at least once.
And the fact that C is "static" and Python is "dynamic" does not make C's type system safe. Heck, even js's type "system" is safer than C's if when you use the word safe you refer to, you know, security.
A language with undefined behavior is not suitable for new code in 2016 if you have alternatives.
Python has undefined behavior because it's implementations are written in languages that do. Every implementation of it is written in a language with undefined behavior such as C, or in languages which are themselves implemented in languages with undefined behavior (e.g. Jython, which runs on the JVM, which is written in C and C++). There is no guarantee that an implementation of Python is bug-free, so there is no guarantee that any Python problem is free of undefined behavior.
So, if you write a Python library in C or in Python, you'll have undefined behavior either way. Undefined behavior is not a good criteria for choosing which language to use.
So, if you write a Python library in C or in Python, you'll have undefined behavior either way. Undefined behavior is not a good criteria for choosing which language to use.
I think you have mixed multiple definitions of "undefined behavior".
A language can say that certain operations are undefined, like reference NULL in C. What of the Python language is undefined? (Some things, like garbage collection, are implementation defined. That is different.)
Even if a language contains undefined behavior, a program written in that language can avoid the undefined operations. What of the Python implementations use or depend on undefined operations in the lower-level language?
An implementation can also be flawed. But that's not "undefined behavior" but non-conformant behavior. "A bug", in the vernacular.
According to your definition, is there any language that doesn't have undefined behavior? After all, even the hardware can have flaws and undefined behavior, so to mix metaphors, it's all a house of cards built upon sand.
A language can say that certain operations are undefined, like reference NULL in C. What of the Python language is undefined? (Some things, like garbage collection, are implementation defined. That is different.)
Even if a language contains undefined behavior, a program written in that language can avoid the undefined operations. What of the Python implementations use or depend on undefined operations in the lower-level language?
An implementation can also be flawed. But that's not "undefined behavior" but non-conformant behavior. "A bug", in the vernacular.
According to your definition, is there any language that doesn't have undefined behavior? After all, even the hardware can have flaws and undefined behavior, so to mix metaphors, it's all a house of cards built upon sand.
> According to your definition, is there any language that doesn't have undefined behavior? After all, even the hardware can have flaws and undefined behavior, so to mix metaphors, it's all a house of cards built upon sand.
That's correct, and it's not pedantic to point that out. It proves that undefined behavior is not a useful criteria for choosing which language to use.
But even if we accept your more restricted definition of what constitutes undefined behavior, we still must accept that any language that is implemented in C could possibly exhibit any of C's undefined behavior. Therefore, writing a Python library in Python instead of C cannot help you avoid C's undefined behavior. It might prevent you from introducing more undefined behavior, but that's not the same as avoiding it entirely -- if the implementation is written in C, then that ship has sailed.
That's correct, and it's not pedantic to point that out. It proves that undefined behavior is not a useful criteria for choosing which language to use.
But even if we accept your more restricted definition of what constitutes undefined behavior, we still must accept that any language that is implemented in C could possibly exhibit any of C's undefined behavior. Therefore, writing a Python library in Python instead of C cannot help you avoid C's undefined behavior. It might prevent you from introducing more undefined behavior, but that's not the same as avoiding it entirely -- if the implementation is written in C, then that ship has sailed.
> That's correct, and it's not pedantic to point that out. It proves that undefined behavior is not a useful criteria for choosing which language to use
No, it means that you've warped the definition of "undefined" to the point where it's no longer useful. It's entirely possible to have fully-defined languages where every possible string of symbols is either a valid program with single well-defined behaviour or not a valid program.
And no, writing the implementation in C does not introduce undefined behaviour, although it may require a number of compile and run time checks to ensure that you're never invoking it.
No, it means that you've warped the definition of "undefined" to the point where it's no longer useful. It's entirely possible to have fully-defined languages where every possible string of symbols is either a valid program with single well-defined behaviour or not a valid program.
And no, writing the implementation in C does not introduce undefined behaviour, although it may require a number of compile and run time checks to ensure that you're never invoking it.
> And no, writing the implementation in C does not introduce undefined behaviour, although it may require a number of compile and run time checks to ensure that you're never invoking it.
This is a valid C program that contains an undefined behavior, from http://blog.regehr.org/archives/213:
#include <limits.h>
#include <stdio.h>
int main (void)
{
It is a perfectly valid program. The outcome is undefined (integer overflow is considered undefined behavior in C by every authoritative source I've ever seen). GCC does not emit any warnings when this program is compiled, even with all warnings turned on. There certainly aren't any runtime checks for it. Any program that adds numbers which are passed into it can run into undefined operations.
Here is another example:
i = i++ + 1;
The result of that operation is undefined.
These examples demonstrate that undefined behavior is not something you "invoke" in a special way -- it's often the result of a mistake. It is not restricted to certain operations -- these examples use simple addition! The compiler and runtime checks can't help you avoid it in many cases -- you sometimes won't even get a compiler warning.
Here is an example of a bug in the C implementation of Python that caused undefined behavior because of integer handling: https://bugs.python.org/issue23999
It's basically impossible to avoid. Even if your code is perfect, the compiler might optimize it into something that might contain undefined behavior. I suppose that if you never did any math, or anything with strings, or any pointer dereferences of any kind, or any casting, or any recursion, and you made sure the compiler didn't try to optimize anything, you could end up with a C program that could not have undefined behavior. But it's clearly impossible to implement Python without doing all of those things.
This is a valid C program that contains an undefined behavior, from http://blog.regehr.org/archives/213:
#include <limits.h>
#include <stdio.h>
int main (void)
{
printf ("%d\n", (INT_MAX+1) < 0);
return 0;
It is a perfectly valid program. The outcome is undefined (integer overflow is considered undefined behavior in C by every authoritative source I've ever seen). GCC does not emit any warnings when this program is compiled, even with all warnings turned on. There certainly aren't any runtime checks for it. Any program that adds numbers which are passed into it can run into undefined operations.
Here is another example:
i = i++ + 1;
The result of that operation is undefined.
These examples demonstrate that undefined behavior is not something you "invoke" in a special way -- it's often the result of a mistake. It is not restricted to certain operations -- these examples use simple addition! The compiler and runtime checks can't help you avoid it in many cases -- you sometimes won't even get a compiler warning.
Here is an example of a bug in the C implementation of Python that caused undefined behavior because of integer handling: https://bugs.python.org/issue23999
It's basically impossible to avoid. Even if your code is perfect, the compiler might optimize it into something that might contain undefined behavior. I suppose that if you never did any math, or anything with strings, or any pointer dereferences of any kind, or any casting, or any recursion, and you made sure the compiler didn't try to optimize anything, you could end up with a C program that could not have undefined behavior. But it's clearly impossible to implement Python without doing all of those things.
You wrote:
> Here is an example of a bug in the C implementation of Python that caused undefined behavior because of integer handling: https://bugs.python.org/issue23999
The comments in that bug report suggest it was a false positive in Coverity.
In any case, the Python language defines how left and right shift are supposed to work. https://docs.python.org/3/reference/expressions.html?highlig... . What you pointed to, if it were a true positive, would be an example of where the implementation didn't comply with the specification.
It wouldn't be an example of undefined behavior as given at https://en.wikipedia.org/wiki/Undefined_behavior : "undefined behavior (UB) is the result of executing computer code that does not have a prescribed behavior by the language specification" because Python prescribes that behavior.
> Here is an example of a bug in the C implementation of Python that caused undefined behavior because of integer handling: https://bugs.python.org/issue23999
The comments in that bug report suggest it was a false positive in Coverity.
In any case, the Python language defines how left and right shift are supposed to work. https://docs.python.org/3/reference/expressions.html?highlig... . What you pointed to, if it were a true positive, would be an example of where the implementation didn't comply with the specification.
It wouldn't be an example of undefined behavior as given at https://en.wikipedia.org/wiki/Undefined_behavior : "undefined behavior (UB) is the result of executing computer code that does not have a prescribed behavior by the language specification" because Python prescribes that behavior.
> It wouldn't be an example of undefined behavior as given at https://en.wikipedia.org/wiki/Undefined_behavior : "undefined behavior (UB) is the result of executing computer code that does not have a prescribed behavior by the language specification" because Python prescribes that behavior.
But the C spec doesn't prescribe any behavior in this case, and the code that is running in this case is C, not Python! It was written in C, it was compiled by a C compiler -- it's C! Whether or not the behavior is undefined for anything a C program does is determined by the C spec only. The Python spec is not relevant. Any C program that overflows an int has done something undefined, whether or not that C program happens to be a Python implementation.
But the C spec doesn't prescribe any behavior in this case, and the code that is running in this case is C, not Python! It was written in C, it was compiled by a C compiler -- it's C! Whether or not the behavior is undefined for anything a C program does is determined by the C spec only. The Python spec is not relevant. Any C program that overflows an int has done something undefined, whether or not that C program happens to be a Python implementation.
The C specification is irrelevant.
The Python specification defines what the implementation is supposed to do. It happens that the implementation doesn't comply with the specification. That doesn't mean the Python specification is undefined, it means the implementation is in error.
Remember too that the actual implementation is a binary. The C compiler converted the C code into that binary, but in theory it could have been generated manually, as a byte-for-byte equivalent, with C completely out of the picture.
Would you still say it's "undefined behavior" if there were no C compiler? How do you tell the difference in the binaries?
The Python specification defines what the implementation is supposed to do. It happens that the implementation doesn't comply with the specification. That doesn't mean the Python specification is undefined, it means the implementation is in error.
Remember too that the actual implementation is a binary. The C compiler converted the C code into that binary, but in theory it could have been generated manually, as a byte-for-byte equivalent, with C completely out of the picture.
Would you still say it's "undefined behavior" if there were no C compiler? How do you tell the difference in the binaries?
> Even if your code is perfect, the compiler might optimize it into something that might contain undefined behavior.
This is false. If your code is perfect, the compiler shouldn't do such optimization. Of course compilers have bugs, but that's compiler bugs not language standard problem. (Not to speak of bug free compilers like CompCert.)
This is false. If your code is perfect, the compiler shouldn't do such optimization. Of course compilers have bugs, but that's compiler bugs not language standard problem. (Not to speak of bug free compilers like CompCert.)
> This is false. If your code is perfect, the compiler shouldn't do such optimization. Of course compilers have bugs
Then it's not false!
Then it's not false!
I now realize I did not understand your original comment. Thank you for the clarification. I don't agree with how you use "undefined behavior", but it's appropriate for the OP's context.
I don't think my definition is different from the common one. I agree that undefined behavior is behavior that the spec for the language says is undefined.
Pretty much every language has some undefined behavior. That undefined behavior can be invoked intentionally, or because of bugs.
C has a lot of undefined behaviors. Even simple addition can result in an undefined behavior because int overflow is undefined. Any C program that takes numbers as input (via the shell, or FFI, or whatever) and adds them together can exhibit undefined behavior. Null pointer dereferences are also undefined, and pretty much any C program of reasonable complexity can encounter those, in the form of bugs.
A bug in hardware can also cause these undefined behaviors to be invoked -- for example, a friend of mine who does embedded programming had some null pointer dereferences or something like that because the hardware did not set a value when he told it to set the value. Null pointer dereferences are undefined, and if they happen because of hardware bugs, that is a case of hardware causing undefined behaviors to happen. It would not be reasonable to say they were not undefined behavior simply because they were caused by hardware instead of programmer error. The fact that it happened at all is what matters, not what caused it.
Even if a language does not have any undefined behaviors whatsoever, that's not the end of it. If a program in that language interacts with other programs that do have undefined behavior, such as C programs, it could potentially trigger or be affected by undefined behaviors in those other programs. If that happens, it's not reasonable to say that it was not undefined behavior simply because it happened in a piece of code in a different language. (So, if the result of running a Python program is affected by undefined behavior in the C implementation of Python, that program has undefined behavior -- even if the Python code does not. The undefined behavior is C's, not Python's -- but it happens because of running the Python code, which caused the C code to run.)
In other words, any invocation of code that causes undefined behavior to happen -- whether it is intentional, or caused by a software bug, or even caused by a hardware bug -- counts.
Pretty much every language has some undefined behavior. That undefined behavior can be invoked intentionally, or because of bugs.
C has a lot of undefined behaviors. Even simple addition can result in an undefined behavior because int overflow is undefined. Any C program that takes numbers as input (via the shell, or FFI, or whatever) and adds them together can exhibit undefined behavior. Null pointer dereferences are also undefined, and pretty much any C program of reasonable complexity can encounter those, in the form of bugs.
A bug in hardware can also cause these undefined behaviors to be invoked -- for example, a friend of mine who does embedded programming had some null pointer dereferences or something like that because the hardware did not set a value when he told it to set the value. Null pointer dereferences are undefined, and if they happen because of hardware bugs, that is a case of hardware causing undefined behaviors to happen. It would not be reasonable to say they were not undefined behavior simply because they were caused by hardware instead of programmer error. The fact that it happened at all is what matters, not what caused it.
Even if a language does not have any undefined behaviors whatsoever, that's not the end of it. If a program in that language interacts with other programs that do have undefined behavior, such as C programs, it could potentially trigger or be affected by undefined behaviors in those other programs. If that happens, it's not reasonable to say that it was not undefined behavior simply because it happened in a piece of code in a different language. (So, if the result of running a Python program is affected by undefined behavior in the C implementation of Python, that program has undefined behavior -- even if the Python code does not. The undefined behavior is C's, not Python's -- but it happens because of running the Python code, which caused the C code to run.)
In other words, any invocation of code that causes undefined behavior to happen -- whether it is intentional, or caused by a software bug, or even caused by a hardware bug -- counts.
Where I disagree with you is your inclusion of language implementation errors as a type of undefined behavior.
Those are compliance errors. That is, if the spec allows it (as "undefined behavior") then it's not a bug. If the spec doesn't allow it, then there is a defined behavior and it is a bug.
For example, a hardware bug is one that does not comply with the specification. Either the spec must be changed (perhaps allowing the existing behavior), or the bug fixed.
Those are compliance errors. That is, if the spec allows it (as "undefined behavior") then it's not a bug. If the spec doesn't allow it, then there is a defined behavior and it is a bug.
For example, a hardware bug is one that does not comply with the specification. Either the spec must be changed (perhaps allowing the existing behavior), or the bug fixed.
Are you saying that when a C program, which happens to be an interpreter for another language, does something that results in undefined behavior according to the C spec, it doesn't count as undefined behavior?
I suspect that is where we differ. It seems to me that if a C program does something that results in undefined behavior according to the C spec, it is undefined behavior period, regardless of any other factors, because the spec says it's undefined and that's all that matters.
I suspect that is where we differ. It seems to me that if a C program does something that results in undefined behavior according to the C spec, it is undefined behavior period, regardless of any other factors, because the spec says it's undefined and that's all that matters.
No, I am not saying that.
I'm saying that when the Python language specification prescribes a behavior, and the Python implementation in C has a different behavior because it does something which the C language specification considers undefined, then the Python implementation in C does not comply with the Python language specification.
I'm saying that it's incorrect to say that non-compliant behavior with respect to the Python specification is the same thing as undefined behavior. It makes no sense to say something is undefined when the specification defines what is supposed to happen.
I'm saying that when the Python language specification prescribes a behavior, and the Python implementation in C has a different behavior because it does something which the C language specification considers undefined, then the Python implementation in C does not comply with the Python language specification.
I'm saying that it's incorrect to say that non-compliant behavior with respect to the Python specification is the same thing as undefined behavior. It makes no sense to say something is undefined when the specification defines what is supposed to happen.
Reports that say that something isn't defined are always interesting to me, because as we define, there are defined undefines; there are things we define we define. We also define there are defined undefines; that is to say we define that there are some things we do not define. But there are also undefined undefines – the ones we don't define we don't define. And if one looks throughout the history of our language and other free languages, it is the latter category that tend to be the non-conformant ones.
> There is no guarantee that an implementation of Python is bug-free, so there is no guarantee that any Python problem is free of undefined behavior.
The implementation being buggy does not mean that the behaviour is undefined in the spec.
The implementation being buggy does not mean that the behaviour is undefined in the spec.
Which spec? The one for Python, or the one for C?
It's entirely possible that executing a Python program could cause C code in the Python interpreter, or C code called via FFI, to do something that the C spec defines as undefined behavior. That could affect the result of the Python program. If that happened, it would not be reasonable to say that the Python program did not have undefined behavior -- even if nothing in the Python code was undefined according to the Python spec.
It's entirely possible that executing a Python program could cause C code in the Python interpreter, or C code called via FFI, to do something that the C spec defines as undefined behavior. That could affect the result of the Python program. If that happened, it would not be reasonable to say that the Python program did not have undefined behavior -- even if nothing in the Python code was undefined according to the Python spec.
How is Python's type system less safe than C's?
It seems the scientific bits of Python have been moving at a much faster development pace though. Do you have any insight as to why?
Personally, I started using Anaconda for everything and just drop down into pip when I find a package that isn't in there already, and it's been working very well.
Personally, I started using Anaconda for everything and just drop down into pip when I find a package that isn't in there already, and it's been working very well.
You should also check out https://conda-forge.github.io/, which is building high quality, community curated conda packages for a lot of stuff that isn't available from the main Continuum channel.
I'm not really commenting on the relative merits of the technology or who should use what. It's about the lack of admission that there is other stuff and other requirements
Hm, I would consider using Anaconda if it worked for Gtk libraries (GEGL etc) on all 3 platforms (yeah I know this is hard outside Anaconda too).
> two packaging tools
wait, what? I always use pip and/or the ubuntu repositories to install scipy. Is there another package manager for it?
wait, what? I always use pip and/or the ubuntu repositories to install scipy. Is there another package manager for it?
I imagine OP is referring to anaconda[0], which is frequently used in scientific contexts and comes with numpy, scipy and other packages included out of the box as well as its own package manager.
0. https://www.continuum.io/
0. https://www.continuum.io/
The actual package distribution/management (it is also a replacement for virtualenv) is conda (http://conda.pydata.org/). Since it can be confusing, the difference between conda, anaconda and miniconda is described here: http://conda.pydata.org/docs/faq.html#general-questions
There is something upsetting about this website, and it may just be that this company is obviously for-profit, but I don't think it needs as much stock photography.
Conda is 100% free and open source, but is developed by Continuum, which is a company (but one that contributes heavily to many of the key components of the open-source pydata/scipy ecosystem, and whose CEO basically created numpy/scipy for the community at the expense of his academic career). Personally I have had nothing but good interactions with people from Continuum and the software they develop. They (as a company and as the individuals in it) do an incredible amount for the open source community and we all benefit from that. They seem to have worked out a sustainable model for running a company and working in/contributing to open source.
Well, this probably isn't the right place to complain about startups that are trying to make money, and I don't have any experience with NumPy, SciPy, or Conda.
I don't know. Do you use Google Chrome, Github, RedHat, LLVM?
Even gcc (egcs) was developed by Cygnus during an important stage.
Even gcc (egcs) was developed by Cygnus during an important stage.
The LLVM foundation is a non-profit and their website is hosted by UIUC.
I prefer programming languages and utilities that are free and open source. And that is some really bad stock photography.
I prefer programming languages and utilities that are free and open source. And that is some really bad stock photography.
Anaconda wants you to do as much as possible via the `conda` tool. Maybe that's what they're referring to.
I think it's referring to the anaconda packages.
Without the scientific stuff, python is just one of a bunch of modern-ish scripting languages out there.
Total tangent: Anyone doing scientific computing work on the JVM may be interested in:
ND4J: N-dimensional arrays for the JVM http://nd4j.org/
Libnd4j: The C++ engine powering the above https://github.com/deeplearning4j/libnd4j
JavaCPP: The bridge between Java and C++ (Cython for Java) https://github.com/bytedeco/javacpp
Fwiw, all that works on Spark with multi-GPUs.
http://deeplearning4j.org/spark
http://deeplearning4j.org/gpu
ND4J: N-dimensional arrays for the JVM http://nd4j.org/
Libnd4j: The C++ engine powering the above https://github.com/deeplearning4j/libnd4j
JavaCPP: The bridge between Java and C++ (Cython for Java) https://github.com/bytedeco/javacpp
Fwiw, all that works on Spark with multi-GPUs.
http://deeplearning4j.org/spark
http://deeplearning4j.org/gpu
Wow, how did I miss javacpp? Looks featureful and actively maintained to boot. Thank you for these recommendations.
Glad you like the looks of it. It was built and is maintained by a Skymind engineer. All of us are on this Gitter channel if you have questions: https://gitter.im/deeplearning4j/deeplearning4j
An example of how, for certain types of code, more than an order-of-magnitude performance improvement can be obtained with a good JIT compiler. I wrote a decompression routine (for a simple compression format) in several languages - it took the following lengths of time to run (normalized to C++ = 1 second):
* CPython, Jython, Lua, MicroPython, Ruby (1.8): >100 seconds
* Cython (naive), IronPython, Ruby (2.3): 50 - 80 seconds
* LuaJIT, PyPy, Cython (with type hints): ~4 seconds
* C#, Go, JavaScript (V8): ~1.5 seconds
* CPython, Jython, Lua, MicroPython, Ruby (1.8): >100 seconds
* Cython (naive), IronPython, Ruby (2.3): 50 - 80 seconds
* LuaJIT, PyPy, Cython (with type hints): ~4 seconds
* C#, Go, JavaScript (V8): ~1.5 seconds
can you try it in julia?
I actually wrote a Julia version at the time, but it was difficult to accurately benchmark because the total runtime was significantly affected by a long (~5 seconds) delay at startup.
Recently, however, I came across https://www.reddit.com/r/Julia/comments/4c09m1/, which suggests running with `--precompiled=yes` to reduce startup time. I have now run the Julia version with that command-line option, and the runtime is impressive - comparable to C#/Go/V8.
Recently, however, I came across https://www.reddit.com/r/Julia/comments/4c09m1/, which suggests running with `--precompiled=yes` to reduce startup time. I have now run the Julia version with that command-line option, and the runtime is impressive - comparable to C#/Go/V8.
Most Julia users run code more often in an interactive REPL session rather than as a script to do a single thing. The precompiled code loading is now on by default on Windows in Julia 0.5 since the backtrace issue was fixed, but there's still a 0.3-0.5 second delay on startup at the moment.
Very cool
I can't help thinking that we need a new scientific programming language that is properly vectorised and directly targets GPUs, as opposed to all these hacks that unroll loops. Now that a large proportion of data science and scientfic computing is GPU centric, aren't there new languages designed with these parallel architectures foremost in mind? Doesn't functional programming map very well to this?
It almost seems to me that just as we're about to get c-speed in Python via JIT tech, so it'll almost immediately be left behind because the GPU is where it's at.
It almost seems to me that just as we're about to get c-speed in Python via JIT tech, so it'll almost immediately be left behind because the GPU is where it's at.
One problem is that functional programming style is often focused on recursion - not just in its functions, but also its data structures. Trees, linked lists and graphs are GPU poison, while arrays map very well to the hardware. However, arrays don't have the pleasant recursive structure that allows you to encode nifty properties (and do pattern matching) the way that is popular in functional programming.
However, array programming is basically functional programming, and does map very well to parallel execution in general. While most array programming is nowadays of the fairly simple form you see in libraries like Numpy, older languages like APL show how far you can go. It's a somewhat alien style of programming to most modern programmers, though.
My rule of thumb: parallel programming is hard. Functional programming makes it trivial to make parallel programs that are correct, but they still might not be fast.
However, array programming is basically functional programming, and does map very well to parallel execution in general. While most array programming is nowadays of the fairly simple form you see in libraries like Numpy, older languages like APL show how far you can go. It's a somewhat alien style of programming to most modern programmers, though.
My rule of thumb: parallel programming is hard. Functional programming makes it trivial to make parallel programs that are correct, but they still might not be fast.
Is Rust the right way forward then? Functional enough without obsessing about linked-memory structures / recursion etc?
I don't think so. Rust has excellent support for concurrency, but does not have any interesting parallel constructs. Which means that while it can probably be fine for coarse-grained multicore parallelism, it does not provide the fine-grain parallelism you need on something like a GPU.
Not built in, but there are libraries, like rayon or crossbeam.
If those recursive structures are static then it's fairly easy to map them into an array form.
There are some attempts, like http://futhark-lang.org/
One problem is that you have to codegen to meager intermediate languages that are fragmented & buggy (OpenCL or GLSL), or go proprietary with just one set of bugs and features (CUDA) but with much limited user base.
Also GPU software stacks are very crashy, have roots in the culture of testing all shaders for driver bugs before shipping in apps ("your game/app is buggy and poorly qa'd if user pc crashes") and only recently has been slightly exposed to dynamic workloads by WebGL (which is only a OpenGL ES 2 subset, impls invest major engineering effort to work around driver bugs, and still had/has big crash problems).
One problem is that you have to codegen to meager intermediate languages that are fragmented & buggy (OpenCL or GLSL), or go proprietary with just one set of bugs and features (CUDA) but with much limited user base.
Also GPU software stacks are very crashy, have roots in the culture of testing all shaders for driver bugs before shipping in apps ("your game/app is buggy and poorly qa'd if user pc crashes") and only recently has been slightly exposed to dynamic workloads by WebGL (which is only a OpenGL ES 2 subset, impls invest major engineering effort to work around driver bugs, and still had/has big crash problems).
Do you think that Vulkan, which also provides a compute API, and appears to be gaining rapid momentum, might solve some of these issues?
Vulkan's SPIR-V might turn out to be a good common target eventually. There's potential but it's all in the execution, is it going to be good enough on majority of platforms? Then there's Apple who are anti-Vulkan so there goes one major platform.
It's also possible that everyone ends up implementing CUDA (AMD, Google, OTOY, PGI have impls out or underway) and it'll get supported as a codegen target.
It's also possible that everyone ends up implementing CUDA (AMD, Google, OTOY, PGI have impls out or underway) and it'll get supported as a codegen target.
OpenCL is fine, but not well supported (because NVIDIA). Vulkan is also fine, so again, the only question is support. I don't know whether Vulkan can supplant OpenCL for computation, or whether it's even supposed to.
None of these solve the problem that the underlying drivers are still terrible (when I test Futhark code I routinely have to restart our GPU servers, as the NVIDIA kernel module crashes). Hopefully, the smaller surface area of OpenCL and Vulkan compared to OpenGL will eventually result in better drivers. I am also very hopeful about AMDs switch to an open source driver model.
None of these solve the problem that the underlying drivers are still terrible (when I test Futhark code I routinely have to restart our GPU servers, as the NVIDIA kernel module crashes). Hopefully, the smaller surface area of OpenCL and Vulkan compared to OpenGL will eventually result in better drivers. I am also very hopeful about AMDs switch to an open source driver model.
Why not a Python to machine code compiler like Cython, but much faster and with all Python 3 capabilities? (e.g. as GNU gcj for Java)
Getting 30x speed up with compiled code using optimized data structures (e.g. pointer compression for better data cache usage, etc.), although not reaching performance of best JIT implementations, it could be good enough.
Getting 30x speed up with compiled code using optimized data structures (e.g. pointer compression for better data cache usage, etc.), although not reaching performance of best JIT implementations, it could be good enough.
The best way forward seems to be to continue to fund these existing JITs. Maybe once one is ready it can replace cpython.
Also pypy not supporting the latest python 3.5 is kind of a big deal for adoption. I would rather have 3.5 support than numpy/cpython compat but I don't do scientific computing :)
Also pypy not supporting the latest python 3.5 is kind of a big deal for adoption. I would rather have 3.5 support than numpy/cpython compat but I don't do scientific computing :)
You ask for it, you got it!
https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from...
https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from...
That is awesome! Python 3.5 all the things :)
Compared this to the Ruby, which is NOT getting much love from the compiler / JIT community.
While one can argue Ruby gets the best and leading edge JIT from Jruby / Graal / Truffle. I dont think it is used much in production. It seems the majority of Python and Ruby has stuck to the default CPython and CRuby runtimes. Both PyPy and JRuby seems to be rather small in usage.
Why may that be?
While one can argue Ruby gets the best and leading edge JIT from Jruby / Graal / Truffle. I dont think it is used much in production. It seems the majority of Python and Ruby has stuck to the default CPython and CRuby runtimes. Both PyPy and JRuby seems to be rather small in usage.
Why may that be?
>LLVM or libraries from Microsoft and IBM can be used to ease the building of JITs.
Which libraries are they talking about? I tried searching but the only mentions of JITs I found on Google were IBM's JVM, and Microsoft's CLR.
Which libraries are they talking about? I tried searching but the only mentions of JITs I found on Google were IBM's JVM, and Microsoft's CLR.
Oracle has GraalVM
http://www.oracle.com/technetwork/oracle-labs/program-langua...
http://www.oracle.com/technetwork/oracle-labs/program-langua...
There's libjit - https://www.gnu.org/software/libjit/
The article mentioned Pyjion, which is Python implemented on top of Microsoft CoreCLR JIT compiler.
Google tried to speed up CPython with a JIT, and failed. Remember Unladen Swallow?
It wasn't a failure because they couldn't do it, they simply ran out of time as they only had a one year mandate and spent the majority of that time fixing bugs in LLVM's JIT.
It's worthwhile to note that LLVM's JIT is now far more mature, and there's been substantial work on GC support.
Back when Unladen Swallow was being worked on, I'm unaware of anything except OS X's OpenGL implementation using LLVM's JIT support—and that ultimately was very different code being JIT'd (they essentially just inline a sequence of interpreter states, so their source IR comes from Clang).
Nowadays… I'm not even going to try listing even dynamically typed language VMs that use LLVM for JITing.
Back when Unladen Swallow was being worked on, I'm unaware of anything except OS X's OpenGL implementation using LLVM's JIT support—and that ultimately was very different code being JIT'd (they essentially just inline a sequence of interpreter states, so their source IR comes from Clang).
Nowadays… I'm not even going to try listing even dynamically typed language VMs that use LLVM for JITing.
LLVM JIT is a lot better now, but it is also quite suboptimal. Even LLVM's fastest code generator is slow compared to other JITs. That's why Apple's JavaScript engine abandoned LLVM JIT: https://webkit.org/blog/5852/introducing-the-b3-jit-compiler...
Its compile-time performance is unquestionably its biggest problem for JITs nowadays. But there's a big difference between being so buggy it's virtually unusable and its performance being suboptimal.
It's also worth pointing out that even "long-running" in JS terms is often considered "short-running" in more general JIT terms, so compile-time performance is even more critical in the JS case than elsewhere.
It's also worth pointing out that even "long-running" in JS terms is often considered "short-running" in more general JIT terms, so compile-time performance is even more critical in the JS case than elsewhere.
IIRC, Unladen Swallow aimed to remove the GIL, which is a different can of worms to open.
Actually Unladen Swallow's aim was simply to be fast. :) They attempted to do that through using LLVM's JIT but ran into a lot of bugs in LLVM and had to spend their time fixing rather than on Python itself. There was no specific work to deal with the GIL.
Iirc Zipy doesn't use a GIL.
Are any of the JIT strategies able to create something that can be invoked as a raw function pointer from C/C++ without any dependency on the GIL?
Not Python, but you might like http://terralang.org/
Numba should be getting this soon - http://numba.pydata.org/numba-doc/dev/proposals/cfunc.html
> Jupyter (formerly IPython) is "an overgrown REPL"
They say that like it's a bad thing.
They say that like it's a bad thing.
How about add JIT and remove GIL at the same time?
Interesting talk to answer part of that question:
https://www.youtube.com/watch?v=P3AyI_u66Bw
There's a third way here - run the C code using the same JIT as the Python code, instead of compiling it natively.
That might sound mind-bending, but C is just a language like any other. It's actually fairly simple and consistent to implement compared to something like Python. You can interpret and JIT compile it if you want to - there's no major magic to that.
Then you can optimise the C code and the Python code at the same time, inline the two, do the same optimisations as you do on Python code, etc. We're using this technique to run Ruby C extensions in JRuby and the results so far are great - running real C extensions faster than native code because we can optimise both the Ruby and the C at the same time.
http://chrisseaton.com/rubytruffle/cext/