Why Python is Slow: Looking Under the Hood (2014)(jakevdp.github.io)
jakevdp.github.io
Why Python is Slow: Looking Under the Hood (2014)
http://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/
182 comments
Exactly. There are a lot of well-designed dynamically typed languages which beat the slow four (python, php, perl, ruby) by a large factor. You are talking about unboxing, but most fast dynamic languages rather have properly tagged primitives. Unboxing is not that performant as starting properly from scratch.
For comparison e.g. php has now proper unboxing optimizations, perl6 also. But the properly tagged primitives such as in Chez scheme, Common Lisp, SELF, Javascript, lua or luajit (to cite the fastest) beat these by at least a factor of 10.
Next problem is refcounting, as you can also easily observe with php7. Getting rid of most local refcounting madness led to a speedup of 100%. A proper GC leads to smaller data, faster updates and memory safety and leakfree-ness.
Next false fact in this article: "Python is interpreted rather than compiled". Nope, cpython has a proper compiler. It translates the AST (a tree) to a linearized list with relative jumps and pretty tight ops and lot of compiler optimizations. Much better than the primitive ad-hoc AST interpreters of php or perl, which still carry around absolute ptrs all over in its ops. The problem with the python bytecode is that is still too big, compared to the fast bytecode interpreters. He probably meant "Python doesn't compile to native code, only bytecode". But there the speedup would be max. 2x, not 10-100 as observed with python. So this argument is bogus.
3. "Python's object model can lead to inefficient memory access" Yes, of course. There's no type optimization done. Which should be trivial and the first thing to do. But more importantly is the design of the hook system. E.g. let's compare it to Common Lisp, where you also can redefine any function/class/object or native data structure access on the fly or Javascript which is similar extremely dynamic. These systems have lots of optimizations for the fast general case, but python not. It generally turns down such optimizations.
For comparison e.g. php has now proper unboxing optimizations, perl6 also. But the properly tagged primitives such as in Chez scheme, Common Lisp, SELF, Javascript, lua or luajit (to cite the fastest) beat these by at least a factor of 10.
Next problem is refcounting, as you can also easily observe with php7. Getting rid of most local refcounting madness led to a speedup of 100%. A proper GC leads to smaller data, faster updates and memory safety and leakfree-ness.
Next false fact in this article: "Python is interpreted rather than compiled". Nope, cpython has a proper compiler. It translates the AST (a tree) to a linearized list with relative jumps and pretty tight ops and lot of compiler optimizations. Much better than the primitive ad-hoc AST interpreters of php or perl, which still carry around absolute ptrs all over in its ops. The problem with the python bytecode is that is still too big, compared to the fast bytecode interpreters. He probably meant "Python doesn't compile to native code, only bytecode". But there the speedup would be max. 2x, not 10-100 as observed with python. So this argument is bogus.
3. "Python's object model can lead to inefficient memory access" Yes, of course. There's no type optimization done. Which should be trivial and the first thing to do. But more importantly is the design of the hook system. E.g. let's compare it to Common Lisp, where you also can redefine any function/class/object or native data structure access on the fly or Javascript which is similar extremely dynamic. These systems have lots of optimizations for the fast general case, but python not. It generally turns down such optimizations.
[deleted]
> (afaik) python is not very jit-friendly in general.
It's not too different from JavaScript, and as I've done some compiler development / run-time code generation myself, I really believe most of the tricks used in the modern JS jit implementations could significantly speed up the current Python. It would be probably too hard to make jits for all the platforms, of course, and it would increase the code base, and surely the memory demands, but for the most popular platforms it would be doable to reach JavaScript jit speeds for the code that does some calculations.
That's what I hoped would have happened at the moment Google took Guido to work for them. But they obviously didn't have such goals.
I think the biggest reason Python is still slow is that there weren't such kind of investments comparable to those for JS jits (or dot Net). On the purely technical level, I really believe it would be very possible. At least for a single-thread execution, and for the heavy calculations, disregarding the famous GIL or some other aspects.
It's not too different from JavaScript, and as I've done some compiler development / run-time code generation myself, I really believe most of the tricks used in the modern JS jit implementations could significantly speed up the current Python. It would be probably too hard to make jits for all the platforms, of course, and it would increase the code base, and surely the memory demands, but for the most popular platforms it would be doable to reach JavaScript jit speeds for the code that does some calculations.
That's what I hoped would have happened at the moment Google took Guido to work for them. But they obviously didn't have such goals.
I think the biggest reason Python is still slow is that there weren't such kind of investments comparable to those for JS jits (or dot Net). On the purely technical level, I really believe it would be very possible. At least for a single-thread execution, and for the heavy calculations, disregarding the famous GIL or some other aspects.
> I think the biggest reason Python is still slow is that there weren't such kind of investments comparable to those for JS jits (or dot Net).
I think this is spot on. Python, Ruby, and the JVM all originated in the 1990-95 timeframe, and all started out with while-loop/switch statement bytecode interpreters. Java had the commercial backing of Sun, who viewed it as a way to compete with Microsoft Windows and invested a lot of money into making Java faster. (HotSpot, etc.) As far as I know, lacking that kind of investment, the main Python and Ruby implementations are still interpreting bytecode 25 years later.
.Net was a little bit different because by the time it came around, JIT was widely understood to be a useful thing, and Microsoft built the CLR with JIT as one of the founding principles.
I also tend to believe that Emacs suffers the same sorts of problem. At it's core, Emacs is a Lisp implementation, and high-performance concurrent language implementations are expensive to build.
I think this is spot on. Python, Ruby, and the JVM all originated in the 1990-95 timeframe, and all started out with while-loop/switch statement bytecode interpreters. Java had the commercial backing of Sun, who viewed it as a way to compete with Microsoft Windows and invested a lot of money into making Java faster. (HotSpot, etc.) As far as I know, lacking that kind of investment, the main Python and Ruby implementations are still interpreting bytecode 25 years later.
.Net was a little bit different because by the time it came around, JIT was widely understood to be a useful thing, and Microsoft built the CLR with JIT as one of the founding principles.
I also tend to believe that Emacs suffers the same sorts of problem. At it's core, Emacs is a Lisp implementation, and high-performance concurrent language implementations are expensive to build.
Yes, some maybe forget, others were too young, but Firefox JavaScript implementation was actually unbelievably, awfully slow for any longer calculations before Google financed and published V8 (2008), e.g. in this benchmark from 2007:
https://www.fourmilab.ch/fourmilog/archives/2007-09/000896.h...
The attitude before the public V8 outside of Google was "it's good enough as it is and it would be too hard to do anything about it." That JS was even slower than Ruby, which was some 30 times slower than C, at that time. Now the jitted JavaScript can run at least at about the speed of the non-optimized C, when doing calculations.
Python is not so slow as that particular JS was then, but I see no technical reason that the speeds comparable to the current JS jits couldn't be reached.
https://www.fourmilab.ch/fourmilog/archives/2007-09/000896.h...
The attitude before the public V8 outside of Google was "it's good enough as it is and it would be too hard to do anything about it." That JS was even slower than Ruby, which was some 30 times slower than C, at that time. Now the jitted JavaScript can run at least at about the speed of the non-optimized C, when doing calculations.
Python is not so slow as that particular JS was then, but I see no technical reason that the speeds comparable to the current JS jits couldn't be reached.
Microsoft's original JavaScript implementation also had some interesting characteristics. Bear with me a moment as I describe why.
Back in 1987, Bill Gates wrote an article for a Byte Magazine special edition in which he described a kind of system-wide scriptability as a next step beyond the simple macro languages common at that time. The general idea is that there would be a single 'macro language' that would have the ability to reach into and automate multiple different applications at the time. Maybe the easiest way to describe his vision from the article is to point to mid-90's VBA. Whatever else you can say about VBA, it represented the culmination of a strategy that Gates had been pushing for 10 years or more. I've always looked to that as an example of the value of long-term thinking in terms of technical strategy. (Even if I do rather dislike the result.)
From the point of view of JavaScript, it's the technology that underpinned VBA's cross-application capabilities that's the most interesting. Sitting at the heart of OLE 2.0/ActiveX was the COM object model, and its late binding support in the form of IDispatch. Without going too far into the details, VBA was able to manipulate COM objects directly. But to do so meant it had to play by COM's rules - reference counting, specific supported data types, etc.
Where JavaScript in the browser comes into the picture is that Microsoft's first JavaScript implementation was just another COM-compatible scripting language, and the IE 6 DOM was exposed as a set of COM objects. This meant that in addition to whatever slowness you had from the JavaScript interpreter, you also had to cope with the COM overhead involved in method calls to DOM objects, as well as the fact that DOM objects in IE6 were reference counted and not traditionally garbage collected. (I was involved in at least one mid-2000's-era project where we spent a bunch of time breaking cycles in DOM/COM object graphs to prevent memory leaks.)
The 'upside' to this approach? Because the DOM was exposed via COM, you could also script your web pages in VBScript.... which was very much in keeping with Microsoft's vision and history, but almost nothing else.
---
One other sidenote, while I'm wandering down memory lane. While OLE _2.0_ was the version that stuck, there was also an OLE _1.0_ that was totally different in implementation. Unlike OLE 2.0, which was based on an actual object model, OLE 1.0 was based on DDE (and shipped in Windows 3.1). DDE was the original MS Windows mechanism for dynamically sharing data across applications, and was heavily based on sending windowing messages from application to application (in the single shared address space common of the time.) IIRC, DDE 1.0 also had the ability to invoke macro commands in other applications, but it was not nearly as developed.
(One of the initial challenges OLE 2.0 faced was that due to the fact it was branded so similarly to the radically inferior OLE 1.0, many people initially missed the value in COM.)
Back in 1987, Bill Gates wrote an article for a Byte Magazine special edition in which he described a kind of system-wide scriptability as a next step beyond the simple macro languages common at that time. The general idea is that there would be a single 'macro language' that would have the ability to reach into and automate multiple different applications at the time. Maybe the easiest way to describe his vision from the article is to point to mid-90's VBA. Whatever else you can say about VBA, it represented the culmination of a strategy that Gates had been pushing for 10 years or more. I've always looked to that as an example of the value of long-term thinking in terms of technical strategy. (Even if I do rather dislike the result.)
From the point of view of JavaScript, it's the technology that underpinned VBA's cross-application capabilities that's the most interesting. Sitting at the heart of OLE 2.0/ActiveX was the COM object model, and its late binding support in the form of IDispatch. Without going too far into the details, VBA was able to manipulate COM objects directly. But to do so meant it had to play by COM's rules - reference counting, specific supported data types, etc.
Where JavaScript in the browser comes into the picture is that Microsoft's first JavaScript implementation was just another COM-compatible scripting language, and the IE 6 DOM was exposed as a set of COM objects. This meant that in addition to whatever slowness you had from the JavaScript interpreter, you also had to cope with the COM overhead involved in method calls to DOM objects, as well as the fact that DOM objects in IE6 were reference counted and not traditionally garbage collected. (I was involved in at least one mid-2000's-era project where we spent a bunch of time breaking cycles in DOM/COM object graphs to prevent memory leaks.)
The 'upside' to this approach? Because the DOM was exposed via COM, you could also script your web pages in VBScript.... which was very much in keeping with Microsoft's vision and history, but almost nothing else.
---
One other sidenote, while I'm wandering down memory lane. While OLE _2.0_ was the version that stuck, there was also an OLE _1.0_ that was totally different in implementation. Unlike OLE 2.0, which was based on an actual object model, OLE 1.0 was based on DDE (and shipped in Windows 3.1). DDE was the original MS Windows mechanism for dynamically sharing data across applications, and was heavily based on sending windowing messages from application to application (in the single shared address space common of the time.) IIRC, DDE 1.0 also had the ability to invoke macro commands in other applications, but it was not nearly as developed.
(One of the initial challenges OLE 2.0 faced was that due to the fact it was branded so similarly to the radically inferior OLE 1.0, many people initially missed the value in COM.)
>Setting typecodes and vals doesn’t slow things down by many orders of magnitude.
What about blowing the cache by having every access involve multiple pointer dereferences?
Virtual calls are absolutely slower than normal calls.
But yeah, 100x is not likely from virtual calls unless you're forcing a cache invalidation on every access. Running a CPU without cache brings you down 100x-200x (literal) in performance on most modern machines.
(The Cell CPU was almost exactly 200, which is why Sony did TONS of PR and tech talks on Data-oriented Design because the Cell would be useless without cache aware programming.)
But on further reflection, you said "this is one source of 100x, not unboxing on its own" so we're more in agreement than disagreement I guess. My point is that virtual calls and dereferences can definitely be a huge performance hit if the cache/TLB can't keep up with it.
What about blowing the cache by having every access involve multiple pointer dereferences?
Virtual calls are absolutely slower than normal calls.
But yeah, 100x is not likely from virtual calls unless you're forcing a cache invalidation on every access. Running a CPU without cache brings you down 100x-200x (literal) in performance on most modern machines.
(The Cell CPU was almost exactly 200, which is why Sony did TONS of PR and tech talks on Data-oriented Design because the Cell would be useless without cache aware programming.)
But on further reflection, you said "this is one source of 100x, not unboxing on its own" so we're more in agreement than disagreement I guess. My point is that virtual calls and dereferences can definitely be a huge performance hit if the cache/TLB can't keep up with it.
> but (afaik) python is not very jit-friendly in general.
It's time for the time honored "Sufficiently Smart Compiler" [0] argument. All programming languages can be optimized in meaningful ways via inferred information. There is a small subset of python programs that I don't see as possible to optimize at all. These programs would be ones using the Pickle library who load code and execute functions from that code. So long as your code is not doing any Evil/Eval() all performance critical portions of your code should theoretically be inferrable.
Note: I said "should be inferrable" NOT "should be easily inferrable". This won't be easy and it takes teams of people to do this on a subset of python's feature set (see Numba).
[0] - http://wiki.c2.com/?SufficientlySmartCompiler
It's time for the time honored "Sufficiently Smart Compiler" [0] argument. All programming languages can be optimized in meaningful ways via inferred information. There is a small subset of python programs that I don't see as possible to optimize at all. These programs would be ones using the Pickle library who load code and execute functions from that code. So long as your code is not doing any Evil/Eval() all performance critical portions of your code should theoretically be inferrable.
Note: I said "should be inferrable" NOT "should be easily inferrable". This won't be easy and it takes teams of people to do this on a subset of python's feature set (see Numba).
[0] - http://wiki.c2.com/?SufficientlySmartCompiler
To my shallow understanding of tracing compilers, eval is not a problem at all since they compile repeating traces (superblocks), not particular function sets nor modules. What you can run you can compile. Iirc, python has too many logic inside vm, so compiling opcodes makes the result no less than ‘efficiently’ calling python internal API, which cannot be optimized since it is written out of vm; compiler has no clue what it does and how to apply transformations on it. To my understanding again, from what I read on topic, if you write a dead simple vm and all higher logic goes to language libraries, i.e. is expressed in the same opcode set, then the runtime will be more explainable to the jit (since all magic will be in trace, explicitly) and optimizations may reduce such stdlibs to series of native instructions not calling any API at all.
This is the case with Lua, where the language has very small and strict opcode set, and probably javascript, which is slightly heavier, but still doesn’t involve too much magic. It is also dummier to my opinion (constant lexical scoping, no green threads, no function environments), so there is a balance between magic, simplicity and money thrown at.
This is the case with Lua, where the language has very small and strict opcode set, and probably javascript, which is slightly heavier, but still doesn’t involve too much magic. It is also dummier to my opinion (constant lexical scoping, no green threads, no function environments), so there is a balance between magic, simplicity and money thrown at.
I’m just gonna jump in.
The JVM has no problem hot loading, unloading, or redefining presently executed code. What I’m trying to say is JIT’ing dynamically loaded code isn’t rocket science
The JVM has no problem hot loading, unloading, or redefining presently executed code. What I’m trying to say is JIT’ing dynamically loaded code isn’t rocket science
It's much more complicated and a much lower on the cost-benefit totem-pole than...
edit: I may be wrong. I'd love to see an example of a VM that implements what you're thinking.
1. Inlining
2. Compiling bytecode for a specific set of input and output types
3. Object Recycling
4. Runtime-compiling to machine code
edit: I may be wrong. I'd love to see an example of a VM that implements what you're thinking.
While the name of the talk implies it's a generally-speaking-talk, Click goes into a lot of detail of how the JVM works in his "A Crash Course in Modern Hardware" presentation[0]. I saw it a while ago, but I think he does mention those things.
Of course that's the one I saw and remember, but you could probably find another presentation in which he goes into much better detail about the JVM and its inner workings.
[0] https://www.youtube.com/watch?v=OFgxAFdxYAQ
Of course that's the one I saw and remember, but you could probably find another presentation in which he goes into much better detail about the JVM and its inner workings.
[0] https://www.youtube.com/watch?v=OFgxAFdxYAQ
Does JavaScript make it easier to reason about those potentially-optimizable areas? Which language features make it easier to optimize to the level that V8 is? (V8 being 7-10x faster than CPython 3 on most of the Benchmarks Game.)
As far as I understand, one important difference is that JavaScript doesn't have __getattr__ or __setattr__ (or at least earlier versions didn't).
You might not use those hooks a lot in your application code, but it seems that web frameworks and the like do a lot of reflection, which makes the code difficult/impossible to optimize (even at runtime with a JIT).
Python also has __add__ (operator overloading) and JavaScript doesn't. This is a good talk about how subtle or crufty the semantics of something like "a+b" is in Python:
https://www.youtube.com/watch?v=IeSu_odkI5I
The PyPy developers had to copy a lot of the implementation details of CPython, which doesn't always result in the fastest code.
You might not use those hooks a lot in your application code, but it seems that web frameworks and the like do a lot of reflection, which makes the code difficult/impossible to optimize (even at runtime with a JIT).
Python also has __add__ (operator overloading) and JavaScript doesn't. This is a good talk about how subtle or crufty the semantics of something like "a+b" is in Python:
https://www.youtube.com/watch?v=IeSu_odkI5I
The PyPy developers had to copy a lot of the implementation details of CPython, which doesn't always result in the fastest code.
You should take a look at the Chambers and Ungar papers on the implementation of Self. In Self, _everything_ is theoretically done through message send/dispatch. This includes field access, numerical operations, flow control... the works. (In a language with prototypical inheritance too.)
What's interesting is that by the time they're done with their optimizations, they manage to get relatively close (x2, iirc) to native speed by essentially inlining everything they can and keeping enough metadata around that they can retain the dynamic properties of the languauge. Impressive stuff. (Which is probably why Sun hired them early in the Java/JIT days.)
What's interesting is that by the time they're done with their optimizations, they manage to get relatively close (x2, iirc) to native speed by essentially inlining everything they can and keeping enough metadata around that they can retain the dynamic properties of the languauge. Impressive stuff. (Which is probably why Sun hired them early in the Java/JIT days.)
I feel like that's more a case of throwing resources at the problem until it improves than the language making life easier for optimizers.
It's at least partially a factor of the cython implementation, because pypy does a pretty great job doing JIT compilation, and making python fast. That's without the resources of Google behind it, too. (Though I've seen some python compiler "20%" style projects roll around: https://github.com/google/grumpy).
That said, python does have a lot of magic functionality that makes predicting paths through the program difficult for a compiler
That said, python does have a lot of magic functionality that makes predicting paths through the program difficult for a compiler
That’s just telling you that a JIT is faster than an interpreter. You’d want to compare PyPy to V8 to see how well they can both be JITed.
Julia is all three, yet it's fast (allowing for jit, which in Julia is a one time cost). It's worth noting that those properties themselves are not what's critical, so much as designing around it and allowing for fast paths through critical code (locking down the dynamic types, first class array datatypes with packed forms)...
It's rather the degree to which Python is dynamic that makes it slow. PyPy could be considered an implicit JIT compiler for Python, yet it is still far slower than Julia. The level of magic you can apply to Python objects that the interpreter/compiler must support is just a different league than Julia. I'd be interested if someone could compare to JS.
Lisp, Smalltalk, Dylan and SELF allow for the same kind of magic.
The JIT developed for SELF is the genesis of Hotspot.
JRuby guys have a quite good implementation making use of Graal, and Ruby is not less magical than Python.
In the end it boils down to how much the community prefers to keep on using C, or improve PyPy.
EDIT: Fixed auto-correction induced typo.
The JIT developed for SELF is the genesis of Hotspot.
JRuby guys have a quite good implementation making use of Graal, and Ruby is not less magical than Python.
In the end it boils down to how much the community prefers to keep on using C, or improve PyPy.
EDIT: Fixed auto-correction induced typo.
s/Gradle/Graal/?
In the case of python it's clear that the heavy reliance on c extensions is a blessing and a curse: it's kept python relevant in communities like science even though it isn't very fast. However one of the lessons of Graal seems to be that such extensions can seriously prohibit improving performance, since they are opaque to JITs.
There's a few talks by Chris Seaton (e.g. https://www.youtube.com/watch?v=YLtjkP9bD_U) on the topic.
In the case of python it's clear that the heavy reliance on c extensions is a blessing and a curse: it's kept python relevant in communities like science even though it isn't very fast. However one of the lessons of Graal seems to be that such extensions can seriously prohibit improving performance, since they are opaque to JITs.
There's a few talks by Chris Seaton (e.g. https://www.youtube.com/watch?v=YLtjkP9bD_U) on the topic.
Yeah, typo due to auto-correction.
If Julia had better support for DataFrames I would jump ship in a heartbeat, but it's just not comparable with R and python/pandas.
Just give it a bit more time to mature.
I'm not sure I'd call Julia a dynamically typed language, each variable can have one and only one (possibly abstract) type which allows the JIT to avoid boxing and opens up plenty of optimisations.
I'm not particularly convinced that Julia's object model leads to inefficient memory access either.
I'm not particularly convinced that Julia's object model leads to inefficient memory access either.
Julia is absolutely dynamically typed [1].
> each variable can have one and only one (possibly abstract) type
This is generally untrue except for a rarely used feature allowing declaration of local variables with a certain type (assignment to such variables automatically converts to the declared type). Is this documented incorrectly somewhere?
Julia's object model allows a spectrum of array types. At one end, `Vector{Any}` is a Python-like array, and at the other, `Vector{Float64}` is a C-like (or NumPy-like) array with a concrete element type. If the element type is abstract but more constrained than `Any` then you get something in between these extremes. This includes array types like `Vector{Union{Float64,Void}}` and `Vector{Real}`, both of which have abstract element types smaller than `Any`. The former can still store values inline like C arrays, thanks to recent work on efficient representation of unions small values. The latter needs Python-like vector-of-pointers representation because the set of kinds of values it can hold is unbounded since you can add new subtypes of `Real` at any time.
[1] https://stackoverflow.com/a/28096079/659248
> each variable can have one and only one (possibly abstract) type
This is generally untrue except for a rarely used feature allowing declaration of local variables with a certain type (assignment to such variables automatically converts to the declared type). Is this documented incorrectly somewhere?
Julia's object model allows a spectrum of array types. At one end, `Vector{Any}` is a Python-like array, and at the other, `Vector{Float64}` is a C-like (or NumPy-like) array with a concrete element type. If the element type is abstract but more constrained than `Any` then you get something in between these extremes. This includes array types like `Vector{Union{Float64,Void}}` and `Vector{Real}`, both of which have abstract element types smaller than `Any`. The former can still store values inline like C arrays, thanks to recent work on efficient representation of unions small values. The latter needs Python-like vector-of-pointers representation because the set of kinds of values it can hold is unbounded since you can add new subtypes of `Real` at any time.
[1] https://stackoverflow.com/a/28096079/659248
> Is this documented incorrectly somewhere?
No, I just got myself massively confused at how code_warntype works. Please just disregard everything I said in the above comment.
No, I just got myself massively confused at how code_warntype works. Please just disregard everything I said in the above comment.
Python is optimized for small programs being easy and quick (and pleasant) to write.
Every other use case it fails in some way. For being slow. For lacking static typing. For lacking compilation. For having a GIL. etc.
Every other use case it fails in some way. For being slow. For lacking static typing. For lacking compilation. For having a GIL. etc.
However, as posters in other thread said, the number and availability of libraries makes functionality of those small programs exceed that of much larger ones written in C or a similar language.
If you have to write a large project from scratch in Python it will be slow, but Python has gone pretty far in the direction where for many large tasks 90% of the functionality will be obtainable through standard libraries. Doing the remaining 10% in basic Python often keeps it competitive at runtime with compiled languages but produce much simpler, shorter and easier to maintain code.
If you have to write a large project from scratch in Python it will be slow, but Python has gone pretty far in the direction where for many large tasks 90% of the functionality will be obtainable through standard libraries. Doing the remaining 10% in basic Python often keeps it competitive at runtime with compiled languages but produce much simpler, shorter and easier to maintain code.
Python has great libraries, but the claim that you can use them and have code that competes with compiled languages is flat out wrong. CPython is at least an order of magnitude slower than Java/Go/Haskell/etc and two orders slower than C/C++/Rust/etc. PyPy closes the gap a bit, but it's still noticeably slower than the aforementioned compiled languages.
Note to responders--HN won't let me respond to you, because "I'm posting too fast".
Note to responders--HN won't let me respond to you, because "I'm posting too fast".
Those statements are too broad to be anything other than flamebait. Which specific code did you benchmark?
That’s not to say that CPython isn’t slower than a compiled language but in common usage the gaps usually end up being much smaller because most of us aren’t paid to run fib() and things like I/O dominate or code ends up being in the same C libraries (libjpeg, OpenSSL, etc.).
I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library.
Again, compiled languages should usually win but the details matter, especially since most people don’t do the exact same work you do. Simply tossing around big numbers with no support contributes noise rather than insight to the conversation.
That’s not to say that CPython isn’t slower than a compiled language but in common usage the gaps usually end up being much smaller because most of us aren’t paid to run fib() and things like I/O dominate or code ends up being in the same C libraries (libjpeg, OpenSSL, etc.).
I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library.
Again, compiled languages should usually win but the details matter, especially since most people don’t do the exact same work you do. Simply tossing around big numbers with no support contributes noise rather than insight to the conversation.
> Which specific code did you benchmark?
I had a csv file with ~16k entries. Each entry has a lat, lon, name and id.
The task: given a (lat, lon) pair, list all entries within 5km.
The simplest straight forward solution is to iterate the list, measure the distance of each item to the given point, and add it to the return list if the distance is within the desired range.
No crazy optimization or anything.
The results were something roughly like this:
The D version runs in 2~4ms when compiled with optimizations turned on to the max, and about 6~8ms without optimizations (just default compiler settings).
The Go version runs in about 6~8ms. (The Go compiler has no option to produce optimized code).
The python version runs in about 50ms.
That's 10x slower than Go and 20x slower than D.
This is python 3. Using pypy shows absolutely no difference in performance.
Obviously the time to load the data from the csv file into a list is not counted in the above results.
> I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library.
That's exactly backwards.
The Go and D versions use structs which are compact data structures that pack the data together. So the entire array fits in one contiguous block of memory.
Python has no concept of structs, so you have to deal with dictionaries.
In general, writing optimized code is not a good idea. You should almost always write the most plain and straight forward code.
Because fast languages are fast by default, there's no optimization required. The straight forward solution is fast enough.
If you are writing Python, your code will be slow by default. You will need to think about ways to optimize it if you want better performance.
I had a csv file with ~16k entries. Each entry has a lat, lon, name and id.
The task: given a (lat, lon) pair, list all entries within 5km.
The simplest straight forward solution is to iterate the list, measure the distance of each item to the given point, and add it to the return list if the distance is within the desired range.
No crazy optimization or anything.
The results were something roughly like this:
The D version runs in 2~4ms when compiled with optimizations turned on to the max, and about 6~8ms without optimizations (just default compiler settings).
The Go version runs in about 6~8ms. (The Go compiler has no option to produce optimized code).
The python version runs in about 50ms.
That's 10x slower than Go and 20x slower than D.
This is python 3. Using pypy shows absolutely no difference in performance.
Obviously the time to load the data from the csv file into a list is not counted in the above results.
> I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library.
That's exactly backwards.
The Go and D versions use structs which are compact data structures that pack the data together. So the entire array fits in one contiguous block of memory.
Python has no concept of structs, so you have to deal with dictionaries.
In general, writing optimized code is not a good idea. You should almost always write the most plain and straight forward code.
Because fast languages are fast by default, there's no optimization required. The straight forward solution is fast enough.
If you are writing Python, your code will be slow by default. You will need to think about ways to optimize it if you want better performance.
I cannot edit my previous comment but just now I revisited the code and noticed there was an import inside the function measuring the distance. One I moved the import out of the function, the execution time was cut by a bit more than half, to about 20ms.
The go version runs in about 3ms, not 6ms.
The D version runs in 2ms.
So overall the python version is 10x slower than native, not 20x slower.
Which is still kind of bad.
The go version runs in about 3ms, not 6ms.
The D version runs in 2ms.
So overall the python version is 10x slower than native, not 20x slower.
Which is still kind of bad.
Worth noting that Python's csv library is optimized C, and it handily beats Go's so this benchmark must be dominated by non-CSV parsing.
The csv parsing is not part of the benchmark.
The only thing I measured is iterating the list and matching entries within range of the input coordinate.
The only thing I measured is iterating the list and matching entries within range of the input coordinate.
People talk about numpy as a numerical library, another way to think of it is as an api for interacting with contiguous chunks of memory.
[deleted]
The whole Ruby VM only takes ~50ms to start, so that sounds a bit slow. I wrote a quick version in Ruby and it only takes 5ms:
https://gist.github.com/jamatthews/d910a2b39c87a871264dd31d1...
It's a bit more complicated. I'm measuring geographical distances so I'm using haversine
from math import radians, cos, sin, asin, sqrt
def distance(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
# taken from: https://stackoverflow.com/a/4913653/35364
# convert decimal degrees to radians
lon1 = radians(lon1)
lon2 = radians(lon2)
lat1 = radians(lat1)
lat2 = radians(lat2)
# haversine formula
dlon = lon2 - lon1
dlat = lat2 - lat1
a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2
c = 2 * asin(sqrt(a))
r = 6371008 # Average radius of earth in meters
return c * rUsing Haversine distance increases it to only 13ms. I'm not sure why your Python implementation is so slow? In the real world, things like haversine are implemented as C extensions, so I wrote one for Ruby as a comparison and it runs in 3.8ms for 16k points. https://gist.github.com/jamatthews/d910a2b39c87a871264dd31d1...
The Ruby GIS libaries already have C implementations for haversine but I just wanted to show how easy it is to hook out to a C function for math heavy stuff.
The Ruby GIS libaries already have C implementations for haversine but I just wanted to show how easy it is to hook out to a C function for math heavy stuff.
It was slow because at first I put the import inside the function. Moving it out improved the python performance to 20ms (as I mentioned in another comment).
The javascript version runs in 15ms.
> In the real world, things like haversine are implemented as C extensions
Having to write C-extensions is, well a point of friction, and adds complexity.
If you just used a native language to begin with, the simple straight forward code would work just fine.
Btw the source is here, and it also includes the csv file. https://bitbucket.org/hasenj/geodb/src
The javascript version runs in 15ms.
> In the real world, things like haversine are implemented as C extensions
Having to write C-extensions is, well a point of friction, and adds complexity.
If you just used a native language to begin with, the simple straight forward code would work just fine.
Btw the source is here, and it also includes the csv file. https://bitbucket.org/hasenj/geodb/src
Yeah, that'll do it!
C extensions are a core part of Ruby. A big chunk of the std lib is implemented in C. You shouldn't be afraid of writing a tiny bit of C. Ruby and Python are scripting languages. They're literally designed for scripting another language.
The JIT for Ruby 3 is well underway, so in a bit it'll be a complete non-issue anyway.
C extensions are a core part of Ruby. A big chunk of the std lib is implemented in C. You shouldn't be afraid of writing a tiny bit of C. Ruby and Python are scripting languages. They're literally designed for scripting another language.
The JIT for Ruby 3 is well underway, so in a bit it'll be a complete non-issue anyway.
> Those statements are too broad to be anything other than flamebait. Which specific code did you benchmark?
> Simply tossing around big numbers with no support contributes noise rather than insight to the conversation.
I resent that you characterize my comment as "flamebait" for lacking supporting evidence for my claim, when I was merely responding to your unsupported claim that Python competes with compiled languages. I chose to be broad because (as you point out), the range varies considerably by application and the margin for error is high, so it would be misleading to be artificially precise. It's perfectly reasonable that you want supporting evidence, but it doesn't make my post "lower quality" than yours.
As for "what I benchmarked", my comment was based on a decade of experience with these languages, including translating many proprietary systems from one language to another (sometimes a faster->slower language and sometimes a slower->faster language depending on whether I was optimizing for performance or security or maintainability).
> That’s not to say that CPython isn’t slower than a compiled language but in common usage the gaps usually end up being much smaller because most of us aren’t paid to run fib() and things like I/O dominate or code ends up being in the same C libraries (libjpeg, OpenSSL, etc.).
> I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library.
I don't dispute that there are cases where Python beats naive C/Go. In particular, Python's csv lib is optimized C code, and it beats Go's CSV parser handily. If all you're doing is parsing CSVs and not actually calling into Python, you should probably stick with Python. While it's true that many applications are I/O bound, my figure was based on CPU time--it wouldn't make sense to judge languages by applications that are inherently I/O bound (although Go's async-by-default is much nicer than Python's async API). Besides, many applications are not I/O bound.
> Simply tossing around big numbers with no support contributes noise rather than insight to the conversation.
I resent that you characterize my comment as "flamebait" for lacking supporting evidence for my claim, when I was merely responding to your unsupported claim that Python competes with compiled languages. I chose to be broad because (as you point out), the range varies considerably by application and the margin for error is high, so it would be misleading to be artificially precise. It's perfectly reasonable that you want supporting evidence, but it doesn't make my post "lower quality" than yours.
As for "what I benchmarked", my comment was based on a decade of experience with these languages, including translating many proprietary systems from one language to another (sometimes a faster->slower language and sometimes a slower->faster language depending on whether I was optimizing for performance or security or maintainability).
> That’s not to say that CPython isn’t slower than a compiled language but in common usage the gaps usually end up being much smaller because most of us aren’t paid to run fib() and things like I/O dominate or code ends up being in the same C libraries (libjpeg, OpenSSL, etc.).
> I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library.
I don't dispute that there are cases where Python beats naive C/Go. In particular, Python's csv lib is optimized C code, and it beats Go's CSV parser handily. If all you're doing is parsing CSVs and not actually calling into Python, you should probably stick with Python. While it's true that many applications are I/O bound, my figure was based on CPU time--it wouldn't make sense to judge languages by applications that are inherently I/O bound (although Go's async-by-default is much nicer than Python's async API). Besides, many applications are not I/O bound.
> I resent that you characterize my comment as "flamebait" for lacking supporting evidence for my claim, when I was merely responding to your unsupported claim that Python competes with compiled languages.
My point was simply that while it's easy to sit around talking about microbenchmarks that's not helpful for most developers. I've seen this cycle repeat so many times over the last couple decades because we're culturally prone to focusing on very small things and ignoring the larger picture:
Most developers are not employed to make a single thing as fast as possible but instead have to ship new features on a regular basis, handle a wide range of inputs correctly, etc.
Most of the performance problems people notice are due to application logic rather than low-level language characteristics, and that tends to mean things like I/O (API calls, poor file access patterns, etc.) or inefficient memory access patterns rather than how fast a loop can be.
I've seen slow applications in compiled languages replaced with a dynamic language and a big performance win not because of the underlying language characteristics but because writing in a lower-level (C, C++, Go) or more convoluted (Java J2EE) environment took enough longer that the development team was struggling just to complete features and the {Perl,PHP,Python,JS} team had considerably more time to optimize the core design.
I've also seen developers in all of those create huge labyrinths which made it hard to reason about the code's performance characteristics, and so my main concern these days is far less caring about the language than the overall amount of complexity you're asking developers to reason about and the related questions about the project and staffing.
Given unlimited time and budget, yes, a compiled language will probably win but for most projects the better question is “what tools will allow me to meet my goals with acceptable performance?” and that's a very different question.
My point was simply that while it's easy to sit around talking about microbenchmarks that's not helpful for most developers. I've seen this cycle repeat so many times over the last couple decades because we're culturally prone to focusing on very small things and ignoring the larger picture:
Most developers are not employed to make a single thing as fast as possible but instead have to ship new features on a regular basis, handle a wide range of inputs correctly, etc.
Most of the performance problems people notice are due to application logic rather than low-level language characteristics, and that tends to mean things like I/O (API calls, poor file access patterns, etc.) or inefficient memory access patterns rather than how fast a loop can be.
I've seen slow applications in compiled languages replaced with a dynamic language and a big performance win not because of the underlying language characteristics but because writing in a lower-level (C, C++, Go) or more convoluted (Java J2EE) environment took enough longer that the development team was struggling just to complete features and the {Perl,PHP,Python,JS} team had considerably more time to optimize the core design.
I've also seen developers in all of those create huge labyrinths which made it hard to reason about the code's performance characteristics, and so my main concern these days is far less caring about the language than the overall amount of complexity you're asking developers to reason about and the related questions about the project and staffing.
Given unlimited time and budget, yes, a compiled language will probably win but for most projects the better question is “what tools will allow me to meet my goals with acceptable performance?” and that's a very different question.
> My point was simply that while it's easy to sit around talking about microbenchmarks that's not helpful for most developers.
To be clear, I never mentioned or implied microbenchmarks.
> Most developers are not employed to make a single thing as fast as possible but instead have to ship new features on a regular basis, handle a wide range of inputs correctly, etc.
I absolutely agree that performance is just factor among many, but the scope of this entire post is "performance". Most of your points seem to pertain to "the biggest performance problem I see is with poor engineering practices". This is definitely a problem, and you're right to prioritize addressing it over choosing a fast language. No language is going to fix stupid, although I will say that Go makes it less appealing to write stupid code, thanks to its restrictive type system. So once you've fixed your stupidity problem, the language-level problems will start to crop up. :)
And by the way, since you mentioned memory access patterns and I/O, I want to point out that these are also language-level concerns. Since everything in Python is a reference into a random location of heap, memory access performance is terrible. Go lets you specify whether something is a pointer or a value, which means you can specify which things are collocated. This is important for good cache performance. Regarding I/O, Python makes you choose between async for performance or sync and your process does no work until the I/O completes. Go lets you write normal synchronous code, and the runtime/scheduler make async calls under the hood so code which doesn't depend on the result of your I/O can execute while other code blocks.
> Given unlimited time and budget, yes, a compiled language will probably win but for most projects the better question is “what tools will allow me to meet my goals with acceptable performance?” and that's a very different question.
It is a very different question, but as new compiled languages improve on ergonomics without compromising performance and as new interpreted languages improve on performance without compromising ergonomics, the answer will increasingly become "not CPython" (Python is the fastest growing language in 2017, but that's largely due to a surge in data science popularity; however, since Python is popular with the data science community largely because of its libraries and not language features, other languages will catch up [and indeed are catching up] quickly).
To be clear, I never mentioned or implied microbenchmarks.
> Most developers are not employed to make a single thing as fast as possible but instead have to ship new features on a regular basis, handle a wide range of inputs correctly, etc.
I absolutely agree that performance is just factor among many, but the scope of this entire post is "performance". Most of your points seem to pertain to "the biggest performance problem I see is with poor engineering practices". This is definitely a problem, and you're right to prioritize addressing it over choosing a fast language. No language is going to fix stupid, although I will say that Go makes it less appealing to write stupid code, thanks to its restrictive type system. So once you've fixed your stupidity problem, the language-level problems will start to crop up. :)
And by the way, since you mentioned memory access patterns and I/O, I want to point out that these are also language-level concerns. Since everything in Python is a reference into a random location of heap, memory access performance is terrible. Go lets you specify whether something is a pointer or a value, which means you can specify which things are collocated. This is important for good cache performance. Regarding I/O, Python makes you choose between async for performance or sync and your process does no work until the I/O completes. Go lets you write normal synchronous code, and the runtime/scheduler make async calls under the hood so code which doesn't depend on the result of your I/O can execute while other code blocks.
> Given unlimited time and budget, yes, a compiled language will probably win but for most projects the better question is “what tools will allow me to meet my goals with acceptable performance?” and that's a very different question.
It is a very different question, but as new compiled languages improve on ergonomics without compromising performance and as new interpreted languages improve on performance without compromising ergonomics, the answer will increasingly become "not CPython" (Python is the fastest growing language in 2017, but that's largely due to a surge in data science popularity; however, since Python is popular with the data science community largely because of its libraries and not language features, other languages will catch up [and indeed are catching up] quickly).
> To be clear, I never mentioned or implied microbenchmarks.
> As for "what I benchmarked", my comment was based on a decade of experience with these languages…
No doubt others have based their comments on their experience, which they trust more.
> As for "what I benchmarked", my comment was based on a decade of experience with these languages…
No doubt others have based their comments on their experience, which they trust more.
I don't disagree; the parent asked where I got my estimates from and I answered.
> Python is the fastest growing language in 2017, but that's largely due to a surge in data science popularity…
How do you know - "that's largely due to"?
How do you know - "that's largely due to"?
There were a lot of analyses that came out about this recently. I don't remember specific articles, but they floated around this and other fora. I think StackOverflow may have had one.
Unless you're using order of magnitude to mean 2x, it's not the case that Java/Go/Haskell are an order of magnitude slower than C/C++/Rust. You can see that on the benchmarks game, the techempower benchmarks, or this lovely paper on optimizing Haskell and C: http://www.leafpetersen.com/leaf/publications/ifl2013/haskel....
I agree that the average is probably around 2X, but the variance and margin for error are pretty large, so I approximated. I didn't mean to imply that there was exactly a tenfold difference.
> Unless you're using order of magnitude to mean 2x, it's not the case that Java/Go/Haskell are an order of magnitude slower than C/C++/Rust.
It depends, in some benchmarks they are 10 times slower.
It depends, in some benchmarks they are 10 times slower.
And you can construct cases where the JVM is infinitely faster than C/C++/Rust, because the JIT optimizes through a virtual call and detects dead code.
Should we say that "it depends, in some cases, Java is infinitely faster than C/C++/Rust"? Not unless we want to deliberately obscure the issue (or we want to warn someone about strange results). Benchmarks are about finding representative results across a variety of tasks.
If you think there are good benchmarks that show these languages are 10x slower in realistic cases, then you should share them, rather than just asserting that they're out there.
Should we say that "it depends, in some cases, Java is infinitely faster than C/C++/Rust"? Not unless we want to deliberately obscure the issue (or we want to warn someone about strange results). Benchmarks are about finding representative results across a variety of tasks.
If you think there are good benchmarks that show these languages are 10x slower in realistic cases, then you should share them, rather than just asserting that they're out there.
> Benchmarks are about finding representative results across a variety of tasks.
> And you can construct cases where the JVM is infinitely faster than C/C++/Rust, because the JIT optimizes through a virtual call and detects dead code.
How would that be a "representative result across a variety of tasks" according to your own definition of a benchmark? What do you mean by "across a variety of tasks"? That sounds like taking the average of some metric for different benchmarks, I'm not sure that is a good idea. I think it makes more sense to benchmark your specific use case, which is what I meant by "it depends".
> If you think there are good benchmarks that show these languages are 10x slower in realistic cases, then you should share them, rather than just asserting that they're out there.
I never said anything about "realistic" and "good", but they are really not unthinkably hard to find: http://benchmarksgame.alioth.debian.org/u64q/binarytrees.htm...
> And you can construct cases where the JVM is infinitely faster than C/C++/Rust, because the JIT optimizes through a virtual call and detects dead code.
How would that be a "representative result across a variety of tasks" according to your own definition of a benchmark? What do you mean by "across a variety of tasks"? That sounds like taking the average of some metric for different benchmarks, I'm not sure that is a good idea. I think it makes more sense to benchmark your specific use case, which is what I meant by "it depends".
> If you think there are good benchmarks that show these languages are 10x slower in realistic cases, then you should share them, rather than just asserting that they're out there.
I never said anything about "realistic" and "good", but they are really not unthinkably hard to find: http://benchmarksgame.alioth.debian.org/u64q/binarytrees.htm...
This is kind of my point; the variance and error margin are pretty wide. Specifying "2x" implies a precision I can't vouch for, so I settled for "an order of magnitude" as the general ballpark. If the 2x figure is important to you, I'm happy to make that concession.
I get the concern, but I'd say "a few times slower", which suggests 2x/3x/4x, not "an order of magnitude", which suggests something like 8x or 20x or whatever.
Orders of magnitude actually suggest the data would be best measured in exponents such as 10^3, 10^4, 10^5, etc because of it's range and distribution.
Can you provide some pointers / examples? This is an honest question -- I am not an expert on bare computation speed, but this (two orders of magnitude vs C++) has not been my experience.
A broad range -- Python programs are "as-fast-as-C" when they are C :-)
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan...
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan...
DEPLOYMENT
My ONE HUGE Gripe with Python. This and that Pandas (I understand why BUT it drove me away) Zero based for statistical work. Your math is 1 based and the language is 0 based.
My ONE HUGE Gripe with Python. This and that Pandas (I understand why BUT it drove me away) Zero based for statistical work. Your math is 1 based and the language is 0 based.
It’s funny because deployment is one of the reasons I hear Python data people trash on R. In general, I’d agree with them that it’s slightly harder for systems people to grok R, but on the other hand I don’t exactly see Python as a shining beacon.
R as a Domain Specific Language has great output for Reports, Word Documents or Power Point slides and a million of reports. There is also the shinny package and shinny server for web publishing.
This is why I went to R from Pandas. My company was impressd with all the reports but asked why they couldn't edit the themes. Then I did it in R in 2 days and haven't looked back. I love Python but I have been using it less and less and more Racket.
This is why I went to R from Pandas. My company was impressd with all the reports but asked why they couldn't edit the themes. Then I did it in R in 2 days and haven't looked back. I love Python but I have been using it less and less and more Racket.
Haha, yes, I keep encountering your comments about R and Racket and have to do a double take to make sure you’re not me.
Would love to hear more about any data Racketeering you’re doing. I’ve been thinking of documenting up my wishlist for quanty things in Racket and maybe try implementing a data dsl in Racket.
Would love to hear more about any data Racketeering you’re doing. I’ve been thinking of documenting up my wishlist for quanty things in Racket and maybe try implementing a data dsl in Racket.
I am thinking of doing a munging and data cleaning dsl and using OpenRefine as a way to implement the code. I really think that would be super handy for doing data cleaning.
A few people have done some R calls within Racket that is intresting.
A few people have done some R calls within Racket that is intresting.
Let me know if you feel like combining forces either for implementation or for imagineering :-)
EDIT: I also know there’s a data frame package called munger, though I it’s undocumented and likely incomplete.
EDIT: I also know there’s a data frame package called munger, though I it’s undocumented and likely incomplete.
Yup! This is what I meant by "lacking compilation".
For example, in Go, you can produce a single statically linked executable and run it on the server directly. No need for a dev-ops/infra team.
For example, in Go, you can produce a single statically linked executable and run it on the server directly. No need for a dev-ops/infra team.
Our Go code is built and tested by Jenkins. If green, the build is stored in an artefact repository. Another job is triggered to update the version in Puppet on potentially multiple targets.
The Puppet module is responsible for getting the binaries from the artefact repository, deploying them with config for the environment, running them as systemd units and shipping the journald output to Elasticsearch.
Static compilation does not obviate devops.
The Puppet module is responsible for getting the binaries from the artefact repository, deploying them with config for the environment, running them as systemd units and shipping the journald output to Elasticsearch.
Static compilation does not obviate devops.
Sounds like you have a lot of unnecessary cruft in your infrastructure.
Deployment and reproducible builds were so bad for us that we switched to Nix. It largely solved the problem, but it has us doing a lot of work to maintain our Nix infrastructure and definitions.
Precisely the opposite reason R drove me away. R uses 1 based indexing.Seriously we should all be using Zero based indexing for all purposes.
One can make an appeal to aesthetics, but that’s no differentiator.
Or one could appeal to adherence to existing standards. Fortran is the oldest (portable) programming language and was 1-based.
Or one could appeal to consistency with the problem domain. Again, math [EDIT: indices are] 1-based.
Zero-based is consistent with C, but that’s about the best argument I can make for zero-based.
Or one could appeal to adherence to existing standards. Fortran is the oldest (portable) programming language and was 1-based.
Or one could appeal to consistency with the problem domain. Again, math [EDIT: indices are] 1-based.
Zero-based is consistent with C, but that’s about the best argument I can make for zero-based.
I think Dijkstra established the case for using Zero based indexing strong enough: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/E...
Maths is zero-based. I'm not sure why you think that numbers start with one; every formulation of the integers in modern maths starts with zero.
It's not. Zero based is wise for offsets (CS) and fields where you have a zero element, and a few corner cases like factorial, but by and large any time you're enumerating an order over a nonempty set you start with one in math, because, max(indices) = |set|
Matrices, common in statistics because of applications of linear algebra, have long maintained the convention of i,j indexes starting with one.
0 based was only for programming for iterations. This makes it a weird math for people. When the first column is 0 and 1 is the second column it is indeed weird and is why statistical languages are 1 based.
This is too categorical. And also it fails to define when small stops being small (and so is easy to defend by moving the goalposts). Yes, these are real problems, but some of them have (partial) solutions. You can use one of the many solutions to write a C, C++, cython and so on extension. You can have type checking with mypy. You can produce executable with cx_freeze (not compilation, but compilation is more of a solution than a problem). You can create parallel programs using multiprocessing. (etc.)
Writing extensions in c/c++ largely negates all stated benefits of using Python in the first place, and I'm proficient in all three languages. Further, mypy is hopeful, but it's not quite there yet (lots of holes and bugs--notably no support for recursive types). Lastly multiprocessing for parallelism is a poor man's solution, and far from what I've become accustomed to in Go, Rust, etc. Python is a nice language and it pays my bills, but Go beats it at many of its own purported strengths, in my opinion.
Using these solution makes the development process much more troublesome than it needs to be, thus defeating the whole point of using python.
Working in digital forensics, I read your comment and immediately thought of Volatility and Rekall, the most important memory forensics frameworks. Both monsters that can analize memory dumps from very different operating systems and microprocessor architectures, supporting a wealth of plugins/extensions to analyze every bit of information you can think of. Both written in Python. Please, tell me how they are failing.
That's largely because Rekall is a fork of Volatility, not because Python was better suited to the task for both projects. Python is used heavily in security/forensics work so it was just a natural choice for the developers.
I'm not sure what your argument is otherwise. If there was a tool written in C or Rust to benchmark against then we could better argue which is better for the task. I don't think anyone said Python was "failing."
I'm not sure what your argument is otherwise. If there was a tool written in C or Rust to benchmark against then we could better argue which is better for the task. I don't think anyone said Python was "failing."
My point is that both of them are big, succesful projects, written in Python, in contrast to hasenjs comment that anything that isn't "a small program being easy and quick (and pleasant) to write... fails in some way.".
Is it really slow? I ported a bunch of code from PHP to Python, nothing incredibly complicated, but it was easy to switch and the Python code all ended up running about 25-40% faster.
I'm sure it would have been even faster to use Java or something like that but it would have taken 5 times as long, and I would have needed to know a lot more Java than Python.
I'm sure it would have been even faster to use Java or something like that but it would have taken 5 times as long, and I would have needed to know a lot more Java than Python.
Yeah, slow is relative. PHP is no one's standard for "fast". I program in Python by day, and I find Go much easier to work with, and it's roughly 10 times faster. There are other performant options besides Java these days.
> PHP is no one's standard for "fast".
PHP is my standard for "fast" in the realm of scripting. Just behind Perl. Python is slower in every prototype benchmark we have ever tried. It's a practical problem that hand waving cannot remedy.
PHP is my standard for "fast" in the realm of scripting. Just behind Perl. Python is slower in every prototype benchmark we have ever tried. It's a practical problem that hand waving cannot remedy.
Fast would be something like Go or D or any of the other compiled languages.
I'm not sure how that relates. None of those are scripting languages, although when you start talking about "compiled" it gets a little messy with definitions. Either way, it's not what I'm talking about.
You're the one who invoked "scripting" out of the blue. There's no reason to artificially limit the conversation to scripting languages, especially since "scripting" is itself poorly defined (what constitutes a scripting language? An interpreter? Where do interpreters end and VMs begin? Python's "interpreter" is a VM; is Python not a scripting language? Python even compiles to Python bytecode--is Python then a compiled language?). If you're going to scope the languages being evaluated, do it by something meaningful, like problem domain--for example, "web applications". I can't speak to D, but Go is certainly a competitor in this space, and speaking from experience, it's much, much more performant than Python and PHP by virtually all metrics. Maybe you mean data science, in which case Python and Go can be compared, but I don't know that anyone is doing real data science work in PHP. Maybe you mean "general purpose language", in which case all of the previously discussed languages can be compared, and in which case Go and D completely outclass Python and PHP.
> You're the one who invoked "scripting" out of the blue.
Not at all. It naturally follows from the incorrect assertion that
> > PHP is no one's standard for "fast".
except within a set of constraints...like scripting languages (of which the pointless bikeshedding often and has occurred).
> There's no reason to artificially limit the conversation to scripting languages,
You've just decided to derail a discussion of a finer point and claim it's in the interest of the main conversation because it's not the main discussion. I can only assume is that you like to argue and had nothing better to do. Good Luck with that.
Not at all. It naturally follows from the incorrect assertion that
> > PHP is no one's standard for "fast".
except within a set of constraints...like scripting languages (of which the pointless bikeshedding often and has occurred).
> There's no reason to artificially limit the conversation to scripting languages,
You've just decided to derail a discussion of a finer point and claim it's in the interest of the main conversation because it's not the main discussion. I can only assume is that you like to argue and had nothing better to do. Good Luck with that.
You introduced an arbitrary, irrelevant constraint as though it was the only valid criterion and got called on it. No need to be sore about it.
what constitutes a scripting language?
It's indeed a bit fuzzy, but program execution without separate compilation (which is not a feature of a language per se) is something I'd look for.
As far as intrinsic language features go, a lack of boilerplate in the sense of
It's indeed a bit fuzzy, but program execution without separate compilation (which is not a feature of a language per se) is something I'd look for.
As far as intrinsic language features go, a lack of boilerplate in the sense of
print("hello")
class Hello {
public static void main(String[] args) {
System.out.println("hello");
}
}
D and Go compilers can both run a script seamlessly without a visible compilation step (it still happens, but behind the scenes, and compilation is fast enough that you would not even notice it).
That's what makes it fuzzy.
I would argue that the defining characteristic of a scripting language is the ability to execute scripts instead of generate binaries. You could then make a further distinctions between application scripting languages (the origin of Lua and Javascript) and general-purpose scripting languages (which are independent of a host application - think Perl or Python).
This distinction is not sharp: Today, Lua and Javascript are used as general-purpose scripting languages, people ship 'fakecutables', packaging script and runtime system into a binary, and even C++ has been used to script applications (physicists are silly people ;)).
But there's normally still a dominant usage pattern associated with a given language, so the classification is not entire useless.
I would argue that the defining characteristic of a scripting language is the ability to execute scripts instead of generate binaries. You could then make a further distinctions between application scripting languages (the origin of Lua and Javascript) and general-purpose scripting languages (which are independent of a host application - think Perl or Python).
This distinction is not sharp: Today, Lua and Javascript are used as general-purpose scripting languages, people ship 'fakecutables', packaging script and runtime system into a binary, and even C++ has been used to script applications (physicists are silly people ;)).
But there's normally still a dominant usage pattern associated with a given language, so the classification is not entire useless.
This isn't a meaningful definition because compilation and execution under a single command is trivial for any compiled language, including Go and Python and PHP. As for lack of boilerplate, Haskell and many statically-typed, compiled functional languages have less boilerplate than PHP or Python. Dynamic typing is more meaningful, but regardless of definition, I fail to see why it's useful to restrict the scope to scripting languages. However, even if you make this restriction, there are many, many languages that are dynamically typed "scripting" languages that are much faster than PHP and Python (JRuby, Lua, JS, Wren, most lisps, etc).
> dynamically typed "scripting" languages that are much faster than PHP and Python (JRuby
?
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan...
?
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan...
That's not the graal backend. The TruffleRuby project uses Graal to make JRuby much, much faster. I don't know what the specific information, but generally "competitive with Java" is my understanding. Perhaps "TruffleRuby" is the name I should have used; it's not apparent and I'm only nominally familiar with the project.
Current Status: "TruffleRuby is progressing fast but is currently probably not ready for you to try running your full Ruby application on."
https://github.com/graalvm/truffleruby#current-status
https://github.com/graalvm/truffleruby#current-status
What are you actually programming? You mention some languages, but you don't actually give any context.
[deleted]
I wonder which version of PHP you were using?
PHP is supposedly faster than Python, nowadays. Not at the JS or LuaJIT level, but it eats CPython and CRuby for breakfast.
I believe this article [0] (previous discussion [1]) from one of pyston's authors gives a much better overview of why python is slow.
[0] http://blog.kevmod.com/2016/07/why-is-python-slow/ [1] https://news.ycombinator.com/item?id=12025309
[0] http://blog.kevmod.com/2016/07/why-is-python-slow/ [1] https://news.ycombinator.com/item?id=12025309
Although I use and enjoy Python for some purposes, I can't help by see all the effort gone into improving Python performance (including Pypy, Cython, rewriting code in C, etc) as fixing a self-inflicted wound. Why are we using a language whose semantics make it very difficult to execute quickly?
Because virtually nothing is bound to anything until we get to this line of code, Python is the ultimate dynamic language.
I find that the vast majority of Python code that I write is not processor-bound or memory-bound. It's disk or network bound (and still would be if I wrote it in C). It's also much simpler to teach programming in than alternatives.
I find that the vast majority of Python code that I write is not processor-bound or memory-bound. It's disk or network bound (and still would be if I wrote it in C). It's also much simpler to teach programming in than alternatives.
Well, if you rewrite in C, you're not using the language whose semantics make it difficult to execute quickly - you're using C. One of the great things about Python is the (relative) ease of interfacing with code written in another language (particularly C), so I don't see this as a square-peg-round-hole situation. You're using the right tool for the job, possibly on the per-function level instead of per-project, but it's much the same.
I really don't see the speed thing as a problem. I write everything in Python until it's slow, then I write some Cython or cuda for the bottleneck. No biggie. One language doesn't need to do everything, and with the particular set of compromises it has made, Python seems to cover a wider range of general-purposesness than most languages, so I'm happy.
I really don't see the speed thing as a problem. I write everything in Python until it's slow, then I write some Cython or cuda for the bottleneck. No biggie. One language doesn't need to do everything, and with the particular set of compromises it has made, Python seems to cover a wider range of general-purposesness than most languages, so I'm happy.
The tooling and ecosystem around it are pretty much unrivalled
From the article: "Dynamic typing makes Python easier to use than C." The author gives no justification for this claim. Do any language experts care to comment?
I'm a big fan of strong static type systems. I believe type-safety increases code quality significantly.
I used to think several years ago, that the main benefit of strong static typing was code safety / eliminating a whole class of bugs. But I've changed my opinion. I now think the biggest benefit is that it makes the code a lot easier for other people to read and understand.
I mean I have multiple personal projects where I've used Python (which is a dynamically typed language), but these are small one-off projects. But I think when working in a team, especially a large team, having types becomes a huge thing. Having types for objects is especially useful. Having types forces you to think more clearly about the structure of your data.
It's really sad when I see `foo(bar)`, and I have no idea what the type of `bar` is, and if it's an object, I have no idea what fields `bar` has. I have to simply guess the structure of the various implicit types by looking at the code (sigh). It makes the code difficult to read, and rather unpleasant to work on. Not to mention, all the multitude of bugs that come from duck/dynamic typing.
I don't think good statically typed languages are hard to use at all. Type inference has spread everywhere that the old argument of having to repeat your types doesn't hold anymore. TypeScript, Flow (JavaScrpt), Haskell, languages from the ML family are really good at type inference. Even the `auto` type inference in C++17 was better than I'd expected.
I used to think several years ago, that the main benefit of strong static typing was code safety / eliminating a whole class of bugs. But I've changed my opinion. I now think the biggest benefit is that it makes the code a lot easier for other people to read and understand.
I mean I have multiple personal projects where I've used Python (which is a dynamically typed language), but these are small one-off projects. But I think when working in a team, especially a large team, having types becomes a huge thing. Having types for objects is especially useful. Having types forces you to think more clearly about the structure of your data.
It's really sad when I see `foo(bar)`, and I have no idea what the type of `bar` is, and if it's an object, I have no idea what fields `bar` has. I have to simply guess the structure of the various implicit types by looking at the code (sigh). It makes the code difficult to read, and rather unpleasant to work on. Not to mention, all the multitude of bugs that come from duck/dynamic typing.
I don't think good statically typed languages are hard to use at all. Type inference has spread everywhere that the old argument of having to repeat your types doesn't hold anymore. TypeScript, Flow (JavaScrpt), Haskell, languages from the ML family are really good at type inference. Even the `auto` type inference in C++17 was better than I'd expected.
Technically Python is now an optionally-typed language, as of version 3.5. It won't actually check the types out-of-the-box, but it has support for annotating variables and functions with their types. A separate project (mypy) can then read and check those types, and it does some inference too.
You can see typed examples here: http://mypy-lang.org/examples.html
You can see typed examples here: http://mypy-lang.org/examples.html
It's still a work in progress though. Buggy, unintuitive, and incomplete (e.g., no recursive types).
> Technically Python is now an optionally-typed language, as of version 3.5
Well no, since the compiler isn't checking types and you need a third party tool for that. So it's not optional typing.
PHP does have run-time optional typing and was successful at retrofitting a static type system in a completely dynamic language.
Mypy is just a static analysis tool.
Well no, since the compiler isn't checking types and you need a third party tool for that. So it's not optional typing.
PHP does have run-time optional typing and was successful at retrofitting a static type system in a completely dynamic language.
Mypy is just a static analysis tool.
Fine, but it's just an import away: https://github.com/kislyuk/ensure (written by HN user ak217)
It does not help you as much if the standard library and the whole ecosystem don't have type annotations.
>Type inference has spread everywhere that the old argument of having to repeat your types doesn't hold anymore.
Indeed. I find one of the most elegant recent examples is this snippet in Kotlin (yes, this is fully and statically typed): https://kotlinlang.org/docs/reference/type-safe-builders.htm...
Indeed. I find one of the most elegant recent examples is this snippet in Kotlin (yes, this is fully and statically typed): https://kotlinlang.org/docs/reference/type-safe-builders.htm...
import com.example.html.*
fun result(args: Array<String>) =
html {
head {
title {+"XML encoding with Kotlin"}
}
body {
h1 {+"XML encoding with Kotlin"}
p {+"this format can be used as an alternative markup to XML"}
// an element with attributes and text content
a(href = "http://kotlinlang.org") {+"Kotlin"}
// mixed content
p {
+"This is some"
b {+"mixed"}
+"text. For more see the"
a(href = "http://kotlinlang.org") {+"Kotlin"}
+"project"
}
p {+"some text"}
// content generated by
p {
for (arg in args)
+arg
}
}
}Count me in. And not only when working in teams. Sometimes that other person is you, after half year or more, and you need to figure out what the hell i was thinking when i was writing this code.
"I used to think several years ago, that the main benefit of strong static typing was code safety / eliminating a whole class of bugs. But I've changed my opinion. I now think the biggest benefit is that it makes the code a lot easier for other people to read and understand."
Obfuscated spaghetti code and write-only programs can be written in any language. I've seen plenty of convoluted OCaml programs, with tons of one-letter variable names and no comments. And don't get me started on Haskell's propensity for using dozens of non-alphabetic operators that make it read more like line noise than Perl, not to mention its famous easy to understand monads or easy to reason about lazyness, on top of the community propensity for one-letter variable names, same as with OCaml. Static typing does not save badly written programs in either of these languages.
Besides, you can add type annotations to your Python programs, and even have them checked by mypy. Those programs might even wind up being easier to read than programs written in a statically-typed language where the programmer relied on type inference. But they won't necessarily be any more readable, as readability has less to do with types than it does with the programmer and the reader.
Obfuscated spaghetti code and write-only programs can be written in any language. I've seen plenty of convoluted OCaml programs, with tons of one-letter variable names and no comments. And don't get me started on Haskell's propensity for using dozens of non-alphabetic operators that make it read more like line noise than Perl, not to mention its famous easy to understand monads or easy to reason about lazyness, on top of the community propensity for one-letter variable names, same as with OCaml. Static typing does not save badly written programs in either of these languages.
Besides, you can add type annotations to your Python programs, and even have them checked by mypy. Those programs might even wind up being easier to read than programs written in a statically-typed language where the programmer relied on type inference. But they won't necessarily be any more readable, as readability has less to do with types than it does with the programmer and the reader.
You're missing the point. Static typing guarantees that you can find out certain very convenient facts about code that you don't fully understand. Which means that if someone writes spaghetti code, you at least have tools that can help you understand it better.
this is exactly my biggest pain-point of dynamic typing (which I do at work): you can't know how a datastructure looks like. Just knowing the fields that are present is something I would like to have.
I thought the most recent Clojure Conj keynote by Rich Hickey (caveat: that guy could sell me the Brooklyn Bridge) did a good job selling dynamically typed languages.
As he makes a pretty good case for, there’s a lot to like about dynamically typed languages. But, everywhere I look in Clojure, I feel like the language is more principled than Python. There’s also Spec, which allows developers to exercise even more rigor without jumping to static types.
Going even further, Racket lets you move to a type system as you want to, but not before.
I’d like to imagine that there’ll be more of these middle ground approaches over time, without having to commit to either the freewheeling status quo of Python or the rigidity of Haskell.
EDIT: I should note that even Python is moving that way as there are now optional type annotations.
As he makes a pretty good case for, there’s a lot to like about dynamically typed languages. But, everywhere I look in Clojure, I feel like the language is more principled than Python. There’s also Spec, which allows developers to exercise even more rigor without jumping to static types.
Going even further, Racket lets you move to a type system as you want to, but not before.
I’d like to imagine that there’ll be more of these middle ground approaches over time, without having to commit to either the freewheeling status quo of Python or the rigidity of Haskell.
EDIT: I should note that even Python is moving that way as there are now optional type annotations.
> It's really sad when I see `foo(bar)`, and I have no idea what the type of `bar` is, and if it's an object, I have no idea what fields `bar` has
With a statically typed language, you still have to look through the code to find the definition of an object. I believe that can be avoided by having easy to access to documentation which would also apply to dynamically typed languages like Python.
With a statically typed language, you still have to look through the code to find the definition of an object. I believe that can be avoided by having easy to access to documentation which would also apply to dynamically typed languages like Python.
Not if you use Hungarian notation!
Shudder.
Shudder.
Right, but modern text editors and IDEs automate the lookup, and failing that, you have the type right there so you can quickly and easily figure out which file contains its definition.
I use Python every day, and the sad (but improving!) state of typing is one of my biggest grievances. Documentation doesn't close the gap because it is often incorrect or outdated; I've added more than a couple production bugs because I trusted the documentation. A type checker is necessary to keep the type annotations in line with the source code.
I use Python every day, and the sad (but improving!) state of typing is one of my biggest grievances. Documentation doesn't close the gap because it is often incorrect or outdated; I've added more than a couple production bugs because I trusted the documentation. A type checker is necessary to keep the type annotations in line with the source code.
> With a statically typed language, you still have to look through the code to find the definition of an object
Static typing (or, more to the point, compile time structure that doesn't require running the code to get right, but the two are closely linked) makes tooling to avoid manual lookups easier to implement and more reliable.
Static typing (or, more to the point, compile time structure that doesn't require running the code to get right, but the two are closely linked) makes tooling to avoid manual lookups easier to implement and more reliable.
>I believe that can be avoided by having easy to access to documentation which would also apply to dynamically typed languages like Python.
staring at five year old project, started as a one-off quick'n'dirty prototype, having gone through many hands doing one-off quick'n'dirty additions since then
What documentation?
But seriously - I don't have to "look through the code". It's a few keystrokes at most and it's always there and correct.
staring at five year old project, started as a one-off quick'n'dirty prototype, having gone through many hands doing one-off quick'n'dirty additions since then
What documentation?
But seriously - I don't have to "look through the code". It's a few keystrokes at most and it's always there and correct.
> you still have to look through the code to find the definition of an object
I haven't done this in the last 10 years. I typically use IDEs, and every IDE I've used has had a "Go To Definition/Declaration" feature, and let you set a key binding for it. On JetBrains' IDEs, I've gotten quite used to pressing Ctrl+B to jump to a type definition's, and then pressing Alt+← to jump back to where I was originally in the code.
I haven't done this in the last 10 years. I typically use IDEs, and every IDE I've used has had a "Go To Definition/Declaration" feature, and let you set a key binding for it. On JetBrains' IDEs, I've gotten quite used to pressing Ctrl+B to jump to a type definition's, and then pressing Alt+← to jump back to where I was originally in the code.
>I typically use IDEs, and every IDE I've used has had a "Go To Definition/Declaration" feature
This is also available in some dynamically typed languages, working just fine.
This is also available in some dynamically typed languages, working just fine.
You can go to the definition, but the definition might just be foo.bar(). Whereas in a statically typed language the IDE can always tell you what the type is (you shouldn't even need to jump to the declaration/definition, you should be able to mouseover or hit a hotkey to just get the type without leaving the line you're on)
Let's imagine a trivial API.
get_product(id)
Or just treat id as whatever and pass it on the service. If it quacks like a duck, it's a duck.
The problem is when it doesn't quack and screams instead.
Yep, but you can catch that and say sorry the id is in the wrong format. Easier to ask for forgiveness then permission.
That's one of the biggest pain points.
You look at code and you have no idea what is what. You have to be on your toes ALL THE TIME!
Even when you are editing a method on a class you cannot safely assume that all the properties will exist because someone can come along and create an instance of the class in a non-standard way such that a property you expect to always exist will not exist in that one situation.
You look at code and you have no idea what is what. You have to be on your toes ALL THE TIME!
Even when you are editing a method on a class you cannot safely assume that all the properties will exist because someone can come along and create an instance of the class in a non-standard way such that a property you expect to always exist will not exist in that one situation.
> someone can come along and create an instance of the class in a non-standard way
shrug just document the interface and stick to it.
If you want to keep things simple, create data structures using namedtuple. You get really sane objects that are definitely POLA [1] compliant.
> You look at code and you have no idea what is what.
You should consider type hints [2] and/or docstrings [2].
[1] https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...
[2] https://docs.python.org/3/library/typing.html
[3] https://docs.python.org/devguide/documenting.html
shrug just document the interface and stick to it.
If you want to keep things simple, create data structures using namedtuple. You get really sane objects that are definitely POLA [1] compliant.
> You look at code and you have no idea what is what.
You should consider type hints [2] and/or docstrings [2].
[1] https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...
[2] https://docs.python.org/3/library/typing.html
[3] https://docs.python.org/devguide/documenting.html
Our project at work is using 3.4 and full of legacy code, and typical python code always has something astonishing.
I always use type hints when I write new code. Because 3.4 has no type hints for class member declarations I end up writing constructors that take parameters then assign them to self. This way PyCharm will properly detect the type of class members.
I always use type hints when I write new code. Because 3.4 has no type hints for class member declarations I end up writing constructors that take parameters then assign them to self. This way PyCharm will properly detect the type of class members.
That's wrong on so many levels.
I rather have the compiler do that for me.
You don't even know whether it quacks or barks or what.
But you need to get the "id as whatever" first.
Let's say you get the id from a HTML form as a string, and want to pass it to the library. Do you need to first convert it to an int? A uuid.UUID object? A custom ID class in the library? Is passing None allowed?
A type declaration on the get_product() function would tell you, and let the compiler to check your code for mistakes.
Let's say you get the id from a HTML form as a string, and want to pass it to the library. Do you need to first convert it to an int? A uuid.UUID object? A custom ID class in the library? Is passing None allowed?
A type declaration on the get_product() function would tell you, and let the compiler to check your code for mistakes.
If they didn't document it, and aren't handling input for an ID that may be passed as an integer, then your problem is with the API, not the language.
And if it quacks like a duck only nine times of ten, you’re outta luck.
I assume less typing, more flexibility from type inference.
Do you mind elaborating? Neither C nor Python have type inference as part of the language, so it's hard to tell exactly what you mean.
Those are type hints, not type inference. Mypy provides some static type checking and inference, aided by those annotations, but mypy is an external tool.
I understand the comment about type inference allowing less typing.
However, I don't understand the OP's statement that type inference provides more flexibility. Any program that type checks under type inference will also type check if the inferred types are made explicit via annotations, so I'm not sure what the OP means by inference providing flexibility, unless they just mean trivial flexibility of surface syntax.
I think the OP may be conflating type inference and gradual/optional typing, but it's hard to be sure from such terse comments.
I understand the comment about type inference allowing less typing.
However, I don't understand the OP's statement that type inference provides more flexibility. Any program that type checks under type inference will also type check if the inferred types are made explicit via annotations, so I'm not sure what the OP means by inference providing flexibility, unless they just mean trivial flexibility of surface syntax.
I think the OP may be conflating type inference and gradual/optional typing, but it's hard to be sure from such terse comments.
The general consensus for the argument where dynamic languages are better and easier to use are
* Safety: Each data carries it's type at run-time, there are no overflow or NULL ptr problems as in statically typed languages. E.g. arithmetic is exact. Memory Safety.
* Easier to develop and debug: You don't have to get right at the first, time, you can iteratively improve the code, e.g. via debugging or refinements, until it works correctly. With statically typed language you have to first make it compile and fix all the errors at once. Generic data structures and methods are much easier to use than specialized.
* Safety: Each data carries it's type at run-time, there are no overflow or NULL ptr problems as in statically typed languages. E.g. arithmetic is exact. Memory Safety.
* Easier to develop and debug: You don't have to get right at the first, time, you can iteratively improve the code, e.g. via debugging or refinements, until it works correctly. With statically typed language you have to first make it compile and fix all the errors at once. Generic data structures and methods are much easier to use than specialized.
There is so much wrong with this...
> Safety: Each data carries it's type at run-time
With a statically typed language, this is not necessary for safety, since the compiler has already checked that the types are correct.
> there are no overflow
Do you mean integer overflow? This is completely orthogonal to the static/dynamic typing distinction.
> or NULL ptr problems as in statically typed languages
This is also completely orthogonal to the static/dynamic typing distinction. Remember that C and C++ do not constitute the entirety of statically typed languages.
> E.g. arithmetic is exact
What does this even mean?
> Memory Safety.
Again, C and C++ do not constitute the entirety of statically typed languages.
> Safety: Each data carries it's type at run-time
With a statically typed language, this is not necessary for safety, since the compiler has already checked that the types are correct.
> there are no overflow
Do you mean integer overflow? This is completely orthogonal to the static/dynamic typing distinction.
> or NULL ptr problems as in statically typed languages
This is also completely orthogonal to the static/dynamic typing distinction. Remember that C and C++ do not constitute the entirety of statically typed languages.
> E.g. arithmetic is exact
What does this even mean?
> Memory Safety.
Again, C and C++ do not constitute the entirety of statically typed languages.
> With a statically typed language, this is not necessary for safety, since the compiler has already checked that the types are correct
Consensus is that static type checking did not help in ensuring memory and concurrency safety (sans the 2 or 3 weird safe languages nobody uses or knows about). The compiler cannot ensure these safeties when the language and VM is too poor to ensure it. I only know ATS, Pony, Idris, Agda and a few more who can ensure proper compile-time safety. Of course static type aficionados are in denial on this point. This is the argument mostly from the LISP community. For the static folks their type system is good enough. Fine for them. Other languages just do it better.
As history has shown dynamically typed languages can easily ensure much more safeties. Esp. arithmetic safeties, with overflows, from int to bigint, from ops to rational, from double to bigfloat, ... This is not at all orthogonal to static/dynamic typing distinction, as the static type systems forbid that. Did you see any union typed arithmetic? Most dynamic languages have that.
> NULL ptr problems completely orthogonal ...
Of course not. I only know a few NULL safe statically typed languages nobody uses. Dynamically typed langs check for these issues in the runtime, thus are safe.
> E.g. arithmetic is exact. What does this even mean?
Oh my. Hope you will never be allowed to work in finance. I know nobody cares about getting numbers right, but speed is not everything. Avoiding underflows and overflows, guaranteeing exact arithmetic with proper type promotions. Ever heard of bigint or rationals? The world doesn't consist only of int, double and decimal. And with e.g. SBCL you have speed and safety.
> Memory Safety. C and C++ ...
But mostly. Add the next 3: Java, Rust, Go from which Rust is also not memory safe.
Consensus is that static type checking did not help in ensuring memory and concurrency safety (sans the 2 or 3 weird safe languages nobody uses or knows about). The compiler cannot ensure these safeties when the language and VM is too poor to ensure it. I only know ATS, Pony, Idris, Agda and a few more who can ensure proper compile-time safety. Of course static type aficionados are in denial on this point. This is the argument mostly from the LISP community. For the static folks their type system is good enough. Fine for them. Other languages just do it better.
As history has shown dynamically typed languages can easily ensure much more safeties. Esp. arithmetic safeties, with overflows, from int to bigint, from ops to rational, from double to bigfloat, ... This is not at all orthogonal to static/dynamic typing distinction, as the static type systems forbid that. Did you see any union typed arithmetic? Most dynamic languages have that.
> NULL ptr problems completely orthogonal ...
Of course not. I only know a few NULL safe statically typed languages nobody uses. Dynamically typed langs check for these issues in the runtime, thus are safe.
> E.g. arithmetic is exact. What does this even mean?
Oh my. Hope you will never be allowed to work in finance. I know nobody cares about getting numbers right, but speed is not everything. Avoiding underflows and overflows, guaranteeing exact arithmetic with proper type promotions. Ever heard of bigint or rationals? The world doesn't consist only of int, double and decimal. And with e.g. SBCL you have speed and safety.
> Memory Safety. C and C++ ...
But mostly. Add the next 3: Java, Rust, Go from which Rust is also not memory safe.
I'm sorry, but I have to disagree with the first point. Static vs dynamic typing doesn't have much to do with safety. Haskell, Standard ML, OCaml, Rust, and many others have memory safety and no null pointer issues. Exact arithmetic and overflow also has nothing to do with type systems. There are statically typed languages other than C and C++, and virtually all of them from the last 20 or more years address the issues you list here.
(I personally would also disagree with your second point, but that is more a matter of opinion.)
(I personally would also disagree with your second point, but that is more a matter of opinion.)
That's an odd consensus since neither exact arbitrary-precision arithmetic nor run-time type information, nor null or memory safety have anything to do with the static/dynamic distinction.
I think there's also no consensus that forcing you to run code to (maybe, if you happen to have chosen the right input values) find errors that could be caught statically really makes it easier to develop.
I think there's also no consensus that forcing you to run code to (maybe, if you happen to have chosen the right input values) find errors that could be caught statically really makes it easier to develop.
I thought this was pretty neat
# WARNNG: never do this!
id113 = id(113)
iptr = IntStruct.from_address(id113)
iptr.ob_digit = 4 # now Python's 113 contains a 4!
113 == 4
ctypes.cast(id113, ctypes.POINTER(ctypes.c_char))[3 * 8] = b'\x71'I agree with this article. In practice, when writing number-crunching code in Python versus Java, I found that Python is usually 10 to 30× slower than Java, sometimes even 100× slower. See: https://www.nayuki.io/page/project-euler-solutions#benchmark...
And yet there are entire industries built on the back of it. Everytime you see a a movie there'll be some Python code somehow responsible for the pixels you're seeing on the big screen.
This is perhaps a great example of why "speed" is a poorly descriptive term for software.
In automobiles, we wouldn't call a large truck "fast" even though it has a much larger (and more "performant") engine than a small car. That small car is likely "much faster" than the truck, and yet cannot do most of the things the truck does.
Sure, python is popular and important, but I don't think that popularity and "speed" are necessarily all that connected at all, except in use-cases where speed is the most important factor (say, financial transactions).
When overnight rendering 3d graphics, speed is important but final product and ease of use are probably more important, since you can compensate for speed with a larger render farm. But more bank server aren't necessarily going to reduce transaction latency (in fact could increase it) so the gains there must often be at a much lower level.
In automobiles, we wouldn't call a large truck "fast" even though it has a much larger (and more "performant") engine than a small car. That small car is likely "much faster" than the truck, and yet cannot do most of the things the truck does.
Sure, python is popular and important, but I don't think that popularity and "speed" are necessarily all that connected at all, except in use-cases where speed is the most important factor (say, financial transactions).
When overnight rendering 3d graphics, speed is important but final product and ease of use are probably more important, since you can compensate for speed with a larger render farm. But more bank server aren't necessarily going to reduce transaction latency (in fact could increase it) so the gains there must often be at a much lower level.
How compatible are existing python libraries with pypy, and is the official python taking clues from pypy? Is there more work to do to make pypy even faster?
Some complex libraries needed special porting (eg. NumPyPy), and there was work under way to get rid of that and provide a CPython compatible interface.
The CPython team does take clues from PyPy (eg: see the CPy3.6 dict implementation), but they are also a lot more careful (some might say "slower") to adopt changes. Both teams also seem to have some ideological differences on how to face the long term development of the language (usualy, GvR goes for "simpler" instead of "more performant").
The CPython team does take clues from PyPy (eg: see the CPy3.6 dict implementation), but they are also a lot more careful (some might say "slower") to adopt changes. Both teams also seem to have some ideological differences on how to face the long term development of the language (usualy, GvR goes for "simpler" instead of "more performant").
Everything I've ever tried with pypy Just Worked. Python packages w/C-extensions have been harder to get right in the past but now PyPy does some excellent C-API emulation.
A thing much bigger than GIL for Python is that in python bytecode, objects are used as primitives.
Except it isn't really. Yes, Python is incredibly slow for day to day stuff, but the sheer amount of easy to use numerical libraries (numpy, scipy, scikit-learn, tensorflow, keras, opencv, just to name a few) make it one of the fastest out there for numerical computation. I tried doing some numerical heavy stuff on the JVM (with Java and Clojure) and it fights you every step of the way ... and that has static typing and all the things the article mentions. Of course, Python derives that functionality from C and Fortran … but just having that interop at your fingertips is magical in terms of productivity. I still get nightmares from working with the JNI.
This point is addressed in literally the first paragraph of the submitted article.
If you read the post, you’d know that you’ve basically summarized it.
The point is not having to write C and Fortran in first place.
As a heavy user of numpy, I use a lot of C and Fortran code, without having to write it.
[deleted]
> Python being a dynamically typed, interpreted language
'CPython' is the defining implementation of the dynamically typed language 'Python' using a byte-code VM (and no jit compiler).
'CPython' is the defining implementation of the dynamically typed language 'Python' using a byte-code VM (and no jit compiler).
bash-3.2$ time /tmp/bench.py
5000000050000000
real 0m19.763s
user 0m15.015s
sys 0m4.309s
bash-3.2$ time /tmp/bench.lisp
5000000050000000
real 0m0.319s
user 0m0.294s
sys 0m0.017s
Yeah, I've never understood that. When I was quite the neophyte I thought interpreted languages couldn't be compiled, and then Common Lisp enlightened me.
This is an unreproducible benchmark. Can we have the programs you used?
See the comments in the original article for the python examples...
The Lisp program is basically this:
But then the SBCL compiler can optimize a type declared version down to 0.07 seconds runtime for the script.
The Lisp program is basically this:
(format t "~a~%" (loop for i from 1 upto 100000000 sum i))
But then the SBCL compiler can optimize a type declared version down to 0.07 seconds runtime for the script.
(locally (declare (optimize (speed 3)
(debug 0)
(safety 0)))
(format t "~a~%" (loop for i fixnum from 1 upto 100000000
sum i of-type fixnum)))> The Python code can be made a lot faster by using xrange and also reduce.
This is a key distinction since it reveals that most of the difference is due to these programs doing different things.
Changing this to compare the same thing shows why this matters:
Since Python 3 made range() lazy by default, the core developers clearly agree that this is better than allocating lists unless explicitly requested.
This is a key distinction since it reveals that most of the difference is due to these programs doing different things.
Changing this to compare the same thing shows why this matters:
cadams@jupiter:~ $ sbcl --version
SBCL 1.4.0
cadams@jupiter:~ $ python2.7 --version
Python 2.7.14
cadams@jupiter:~ $ pypy --version
Python 2.7.13 (84a2f3e6a7f88f2fe698e473998755b3bd1a12e2, Oct 05 2017, 16:34:13)
[PyPy 5.9.0 with GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]
cadams@jupiter:~ $ time ./test.lisp
5000000050000000
real 0m0.209s
user 0m0.194s
sys 0m0.011s
cadams@jupiter:~ $ time python2.7 test.py
5000000050000000
real 0m0.758s
user 0m0.744s
sys 0m0.008s
cadams@jupiter:~ $ time pypy test.py
5000000050000000
real 0m0.123s
user 0m0.101s
sys 0m0.019s
Since Python 3 made range() lazy by default, the core developers clearly agree that this is better than allocating lists unless explicitly requested.
Not exactly. Setting typecodes and vals doesn’t slow things down by many orders of magnitude. The main reason python is relatively slow is that there is no practical way to reason about what parts of program may be optimized out or leveled down to native datatypes and then restructured in a very efficient way. This is what optimizing/jit compilers do to achieve much performance; this one is the source of x100, not unboxing on its own. Technically, tracing jit that doesn’t care if you’re writing in static or dynamic, strict or duck typing can be done for any language, but (afaik) python is not very jit-friendly in general.