I Want a New Duck (2020)(glyph.twistedmatrix.com)
glyph.twistedmatrix.com
I Want a New Duck (2020)
https://glyph.twistedmatrix.com/2020/07/new-duck.html
122 comments
At this point, another big benefit of Python is its huge number of libraries. This is really attractive to lots of people who also want static type checking.
Oftentimes, the benefits of using Python in a code base that would benefit from static typing outweigh the costs. Especially when tools like MyPy exist, which aren't perfect but help tremendously.
Oftentimes, the benefits of using Python in a code base that would benefit from static typing outweigh the costs. Especially when tools like MyPy exist, which aren't perfect but help tremendously.
This can't be emphasized enough. I'm presently pushing mypy and type hints hard in a Python project (passing static analysis and type checks is required to get through the CI pipeline) and it's just a compromise. Honestly, I don't want a dynamic duck-typed language at all, but it is what it is. Right now, the single most important factor determining whether we have any future at all is speed to market, not the increased reliability you get from type safety. And Python just has libraries for basically everything. It's enabled us to spin up brand new services that do exactly what we need to do from scratch in a matter of hours sometimes.
With competing priorities like that, this is the best we can do right now. The other factor is the rest of the team is already familiar with Python. They're mostly infrastructure developers. They're more likely to do everything in Bash if given the choice. I'm not going to teach them Scala or Haskell or Rust in the next three weeks before we need to deliver something. But I can at least teach them Python's optional type hints and mypy.
With competing priorities like that, this is the best we can do right now. The other factor is the rest of the team is already familiar with Python. They're mostly infrastructure developers. They're more likely to do everything in Bash if given the choice. I'm not going to teach them Scala or Haskell or Rust in the next three weeks before we need to deliver something. But I can at least teach them Python's optional type hints and mypy.
Projects tend to change over their lifetime. What might have been yesterday's freewheeling experimental prototype is tomorrow's boring mission-critical geriatric legacy. Sometimes this is (or should be) known in advance, but often not.
Which is why recent language design has converged on sound typing augmented by powerful type inference, so that you can use the same language at any scale. I've worked on ten-line database migration scripts and million-line applications in the same Scala.
As I have in Python. It has issues at the high end, but with tooling they can mostly be overcome. It could be better, but I'm not complaining.
On the other had we have: https://news.ycombinator.com/item?id=26539508
Sounds like not everything in Scalaland is sunshine and roses.
On the other had we have: https://news.ycombinator.com/item?id=26539508
Sounds like not everything in Scalaland is sunshine and roses.
Sometimes you know in advance, but you still want to optimize for starting and not for maintaining.
>I can't help but feel like if you want static typing, you're better off recognizing that python is not the right tool for the job.
Static typing is not a job though. It's a language feature. The job is solving technical/business problems.
And Python is a tool, but it's not a tool to do dynamic typing or static typing with, it's a tool to solve problems with.
So, the comment "if you want static typing, you're better off recognizing that python is not the right tool for the job" doesn't quite sit right.
You could instead better say that it's not a good fit for Python.
But why would that be?
Especially since one could easily have said the same for Javacript, but Typescript exists as basically Javascript + types, and people seem to love it.
So there's that.
Static typing is not a job though. It's a language feature. The job is solving technical/business problems.
And Python is a tool, but it's not a tool to do dynamic typing or static typing with, it's a tool to solve problems with.
So, the comment "if you want static typing, you're better off recognizing that python is not the right tool for the job" doesn't quite sit right.
You could instead better say that it's not a good fit for Python.
But why would that be?
Especially since one could easily have said the same for Javacript, but Typescript exists as basically Javascript + types, and people seem to love it.
So there's that.
Typescript a) is just plain a lot better than mypy b) is still a poor experience compared to using a first-class typed language.
The extent to which a) is due to implementation decisions made by typescript/mypy vs being due to inherent differences between Javascript and Python is arguable. Certainly there are things that look like unforced design errors in mypy, and the fact that Dart existed (and largely failed) before Typescript shows that it's not just about what language you're based on. But there are also idioms and aspects of the Python object model that seem inherently hard to type nicely, and are sadly too entrenched in the ecosystem to change.
The extent to which a) is due to implementation decisions made by typescript/mypy vs being due to inherent differences between Javascript and Python is arguable. Certainly there are things that look like unforced design errors in mypy, and the fact that Dart existed (and largely failed) before Typescript shows that it's not just about what language you're based on. But there are also idioms and aspects of the Python object model that seem inherently hard to type nicely, and are sadly too entrenched in the ecosystem to change.
> Typescript (...) is still a poor experience compared to using a first-class typed language.
I personally don't agree. I have coded in C# and TS extensively, and while first-class types available in runtime (especially with generics <cough>though not in java</cough>) is super nice, I think the benefits of looseness of TS overweight the costs. Also, you can always go crazy and use zod or io-ts, but in that road there's always the danger of just writing types and not doing any work because "typing is fun"(c).
I personally don't agree. I have coded in C# and TS extensively, and while first-class types available in runtime (especially with generics <cough>though not in java</cough>) is super nice, I think the benefits of looseness of TS overweight the costs. Also, you can always go crazy and use zod or io-ts, but in that road there's always the danger of just writing types and not doing any work because "typing is fun"(c).
I spent a couple of days trying to make an app work in typescript before thinking "I'll just give Scala.js a quick try, not going to spend a lot of time on it", and to my surprise everything just worked perfectly.
I'm happy to hear that, even though I'd personally refrain from getting myself into a Scala codebase again :) (IMHO, it goes further in the "typing for typing's sake" direction). Because of my existing skills I'd probably use Blazor, if I'm going that way.
Javascript has no real competition. People have accepted its existence as a given so every improvement over it is celebrated. On the other hand, many languages want to be the new python and their fans will try to bring it down at every opportunity.
Note: I'm a python dev, yet even I can agree that the python ecosystem is broken and if not maintained, the language will decline over time.
Note: I'm a python dev, yet even I can agree that the python ecosystem is broken and if not maintained, the language will decline over time.
What's the equivalent for numpy? Tensorflow? Pytorch? The python library support is simply unmatched: regardless of what the base language has as features, the proper choice for many projects is python.
I think that consideration of libraries is a good point, I didn't initially consider, and especially relevant for Python.
I think the one connection I'd make back to my original point is that perhaps python becoming the defacto interface for certain libraries is still a reflection of its core values.
This is especially evident with libraries like Tensorflow, which have interfaces for a breadth of languages, and which the core is implemented in C++. The reason people tend to reach for python to call into Tensorflow is still the core platform values of ease of use, rapid prototyping (imo).
I think the one connection I'd make back to my original point is that perhaps python becoming the defacto interface for certain libraries is still a reflection of its core values.
This is especially evident with libraries like Tensorflow, which have interfaces for a breadth of languages, and which the core is implemented in C++. The reason people tend to reach for python to call into Tensorflow is still the core platform values of ease of use, rapid prototyping (imo).
> I can't help but feel like if you want static typing, you're better off recognizing that python is not the right tool for the job.
If the one and only thing you want is static typing, sure.
That's, like, never the case in programming, so it's not really meaningful, though.
In real world programming scenarios, you are always dealing with balancing multiple dimensional problems and “python, with some degree of static typing” is a reasonable component of the solution for lots of them.
> Or said differently, adding mypy feels like trying to get out of a trade-off decision.
No, considering python’s available static type checking (whether mypy, pyright, or whatever, or even a combination) is turning a big coarse trade-off decision into one with finer-grained options, not avoiding it.
> For me, platform core values that stick out for python would be Approachability, Simplicity, and Velocity. I understand the posited value mypy brings to the table, but it feels in contention with the original core values that made python appealing to begin with.
IME using Python’s typing, usually incrementally added, it's not particularly. In fact, it's an enhancer to velocity as the code base evolves.
If the one and only thing you want is static typing, sure.
That's, like, never the case in programming, so it's not really meaningful, though.
In real world programming scenarios, you are always dealing with balancing multiple dimensional problems and “python, with some degree of static typing” is a reasonable component of the solution for lots of them.
> Or said differently, adding mypy feels like trying to get out of a trade-off decision.
No, considering python’s available static type checking (whether mypy, pyright, or whatever, or even a combination) is turning a big coarse trade-off decision into one with finer-grained options, not avoiding it.
> For me, platform core values that stick out for python would be Approachability, Simplicity, and Velocity. I understand the posited value mypy brings to the table, but it feels in contention with the original core values that made python appealing to begin with.
IME using Python’s typing, usually incrementally added, it's not particularly. In fact, it's an enhancer to velocity as the code base evolves.
Python is one of my least favorite languages, but it is a language I absolutely have to use for data work – it would be impractical to rewrite everything in Julia.
Type annotations help a lot. They're not perfect, but for long-running jobs it's a huge help to catch something when PyCharm highlights it instead of 30 minutes into the job.
Type annotations help a lot. They're not perfect, but for long-running jobs it's a huge help to catch something when PyCharm highlights it instead of 30 minutes into the job.
I'm curious to know what makes it one of your least favourite languages. Could you elaborate?
The reason I'm interested is because I love python and would like to hear differing opinions
The reason I'm interested is because I love python and would like to hear differing opinions
Sure! Broadly speaking, I think Python emphasizes a combination of features that lead to code that isn't very composable or decoupled, but without the error-checking benefits you might get in a language like Scala.
For example, consider implementing complex rational numbers. If you have a complex number type, and a rational number type, it should be relatively easy to combine them. Julia does this with literally 11 lines of code in Rational.jl, and 0 lines of code in Complex.jl . In Python, this is extremely difficult, to the point that nobody does it – you would need to restructure the class hierarchy, and you can't do that without access to both.
Another example is the lack of extension methods. For example, if you want to add a bit of functionality to a string or a sqlalchemy engine, you can either add functions (which have a different syntax) or inherit and follow the awful "MyString" pattern.
Python chose to emphasize list comprehensions over map + filter. The problem with that is that you now have syntax that doesn't generalize to other collections, especially custom collections.
Some patterns like async/await, decorators, and with syntax encourage hardcoding decisions when writing a function, as opposed to when using it. This makes your functions less flexible and means you have to write more of them. E.g. consider Julia's do-block syntax[1], which is very similar to Python's with but based on function composition and far more general as a result. Or compare Julia's @spawn to Python's async/await.
[1]: https://docs.julialang.org/en/v1/manual/functions/#Do-Block-...
For example, consider implementing complex rational numbers. If you have a complex number type, and a rational number type, it should be relatively easy to combine them. Julia does this with literally 11 lines of code in Rational.jl, and 0 lines of code in Complex.jl . In Python, this is extremely difficult, to the point that nobody does it – you would need to restructure the class hierarchy, and you can't do that without access to both.
Another example is the lack of extension methods. For example, if you want to add a bit of functionality to a string or a sqlalchemy engine, you can either add functions (which have a different syntax) or inherit and follow the awful "MyString" pattern.
Python chose to emphasize list comprehensions over map + filter. The problem with that is that you now have syntax that doesn't generalize to other collections, especially custom collections.
Some patterns like async/await, decorators, and with syntax encourage hardcoding decisions when writing a function, as opposed to when using it. This makes your functions less flexible and means you have to write more of them. E.g. consider Julia's do-block syntax[1], which is very similar to Python's with but based on function composition and far more general as a result. Or compare Julia's @spawn to Python's async/await.
[1]: https://docs.julialang.org/en/v1/manual/functions/#Do-Block-...
> For example, consider implementing complex rational numbers. If you have a complex number type, and a rational number type, it should be relatively easy to combine them. Julia does this with literally 11 lines of code in Rational.jl, and 0 lines of code in Complex.jl . In Python, this is extremely difficult, to the point that nobody does it
It's really not difficult; yes, it's more verbose, sure.
> Python chose to emphasize list comprehensions over map + filter. The problem with that is that you now have syntax that doesn't generalize to other collections, especially custom collections.
Python has map/filter/reduce and they work fine. Comprehensions (and genexps) are more concise and cleae for 95% of use cases, I find, so I think the right choice was made there, though I do think people do tend to reach for them sometimes when map, filter, and friends are more appropriate.
> consider Julia’s do-block syntax[1], which is very similar to Python's with
It's more similar to Ruby’s block notation than Python with; it constructs an anonymous function and passes it to another function being called. It is true that with multiline lambdas at all, much less a convenience notation for passing them to other functions, with notation would be superfluous.
> Or compare Julia’s @spawn to Python’s async/await.
Why? They solve different problems. @spawn is equivalent to Python’s concurrent.futures.submit, and it’s buddy fetch() is concurrent.futures.Future.result(). Python has had those longer than it has async/await.
It's really not difficult; yes, it's more verbose, sure.
> Python chose to emphasize list comprehensions over map + filter. The problem with that is that you now have syntax that doesn't generalize to other collections, especially custom collections.
Python has map/filter/reduce and they work fine. Comprehensions (and genexps) are more concise and cleae for 95% of use cases, I find, so I think the right choice was made there, though I do think people do tend to reach for them sometimes when map, filter, and friends are more appropriate.
> consider Julia’s do-block syntax[1], which is very similar to Python's with
It's more similar to Ruby’s block notation than Python with; it constructs an anonymous function and passes it to another function being called. It is true that with multiline lambdas at all, much less a convenience notation for passing them to other functions, with notation would be superfluous.
> Or compare Julia’s @spawn to Python’s async/await.
Why? They solve different problems. @spawn is equivalent to Python’s concurrent.futures.submit, and it’s buddy fetch() is concurrent.futures.Future.result(). Python has had those longer than it has async/await.
> It's really not difficult; yes, it's more verbose, sure.
Can you point to a case where it's actually been done? I don't even see how you would do it in Python without modifying the standard library or reimplementing significant chunks of cmath.py or fractions.py . In languages like Julia or Haskell this is possible (and relatively easy) as a third party who's just importing the two types.
> Python has map/filter/reduce and they work fine.
Python's map has a few problems, e.g. crippled lambdas make it less useful and it returns a map object instead of the same type as the original collection. It's better than nothing, but it's noticeably weaker than map in other languages.
> Comprehensions (and genexps) are more concise and cleae for 95% of use cases, I find, so I think the right choice was made there, though I do think people do tend to reach for them sometimes when map, filter, and friends are more appropriate.
One big use case where it doesn't apply is numpy arrays or pytorch tensors. It's very common, at least in my domain, for people to initially write code with lists and then switch to arrays for performance. But despite the fact that they should have mostly the same interface, this requires lots of little syntax changes and it's easy to introduce a bug.
Haskell has this issue too to some extent, linked lists are the default data structure and `map` doesn't work with others, you need `fmap`.
> It's more similar to Ruby’s block notation than Python with; it constructs an anonymous function and passes it to another function being called. It is true that with multiline lambdas at all, much less a convenience notation for passing them to other functions, with notation would be superfluous.
Good point.
> Why? They solve different problems. @spawn is equivalent to Pythons submit from concurrent.futures, and it's buddy fetch() is Future.result(). Python has had that longer than async/await.
I do like concurrent.futures, but most blog posts, documentation etc. recommend using asyncio when either one would work, and the majority of libraries at this point use asyncio. I don't think it's really possible to avoid asyncio at this point.
Can you point to a case where it's actually been done? I don't even see how you would do it in Python without modifying the standard library or reimplementing significant chunks of cmath.py or fractions.py . In languages like Julia or Haskell this is possible (and relatively easy) as a third party who's just importing the two types.
> Python has map/filter/reduce and they work fine.
Python's map has a few problems, e.g. crippled lambdas make it less useful and it returns a map object instead of the same type as the original collection. It's better than nothing, but it's noticeably weaker than map in other languages.
> Comprehensions (and genexps) are more concise and cleae for 95% of use cases, I find, so I think the right choice was made there, though I do think people do tend to reach for them sometimes when map, filter, and friends are more appropriate.
One big use case where it doesn't apply is numpy arrays or pytorch tensors. It's very common, at least in my domain, for people to initially write code with lists and then switch to arrays for performance. But despite the fact that they should have mostly the same interface, this requires lots of little syntax changes and it's easy to introduce a bug.
Haskell has this issue too to some extent, linked lists are the default data structure and `map` doesn't work with others, you need `fmap`.
> It's more similar to Ruby’s block notation than Python with; it constructs an anonymous function and passes it to another function being called. It is true that with multiline lambdas at all, much less a convenience notation for passing them to other functions, with notation would be superfluous.
Good point.
> Why? They solve different problems. @spawn is equivalent to Pythons submit from concurrent.futures, and it's buddy fetch() is Future.result(). Python has had that longer than async/await.
I do like concurrent.futures, but most blog posts, documentation etc. recommend using asyncio when either one would work, and the majority of libraries at this point use asyncio. I don't think it's really possible to avoid asyncio at this point.
> I don’t even see how you would do it in Python without modifying the standard library or reimplementing significant chunks of cmath.py or fractions.py
If I understand the problem correctly, and you are trying to implement Gaussian Rationals, you’d inherit from numbers.Complex, storing the real and imaginary parts as instances of numbers.Rational (you could use the concrete fractions.Fraction, but I don’t think you actually would need to reference the concrete type to do the implementation.)
There’s a bit of boilerplate isinstance-stuff implementing the operations, which is somewhat tedious, but not difficult (I think its less involved than the custom integral class used as an example in the numbers module documentation, because you should be able to get by only handling same-class, Rational, and Complex cases for most ops.
> One big use case where it doesn’t apply is numpy arrays or pytorch tensors.
Good point. It is not one I’ve hit a lot because of the stuff most of python code use is on doesn’t hit those, but I’ve seen that.
> I do like concurrent.futures, but most blog posts, documentation etc. recommend using asyncio when either one would work, and the majority of libraries at this point use asyncio. I don’t think it’s really possible to avoid asyncio at this point.
If I understand the problem correctly, and you are trying to implement Gaussian Rationals, you’d inherit from numbers.Complex, storing the real and imaginary parts as instances of numbers.Rational (you could use the concrete fractions.Fraction, but I don’t think you actually would need to reference the concrete type to do the implementation.)
There’s a bit of boilerplate isinstance-stuff implementing the operations, which is somewhat tedious, but not difficult (I think its less involved than the custom integral class used as an example in the numbers module documentation, because you should be able to get by only handling same-class, Rational, and Complex cases for most ops.
> One big use case where it doesn’t apply is numpy arrays or pytorch tensors.
Good point. It is not one I’ve hit a lot because of the stuff most of python code use is on doesn’t hit those, but I’ve seen that.
> I do like concurrent.futures, but most blog posts, documentation etc. recommend using asyncio when either one would work, and the majority of libraries at this point use asyncio. I don’t think it’s really possible to avoid asyncio at this point.
When programmers have learned a particular language really well and they encounter a problem the language does not handle well it is usual for them to want to add some capability to the language to handle their problem rather than going off and learning a new language, often because the constraints of time does not let them learn the new language.
There is a tendency to think of pragmatism as throwing away core values in order to solve a problem, although generally this is in the use of the word pragmatism in the domains of business or politics and not in the domain of programming language extensibility.
There is a tendency to think of pragmatism as throwing away core values in order to solve a problem, although generally this is in the use of the word pragmatism in the domains of business or politics and not in the domain of programming language extensibility.
I can't talk about python, but it's convenient to be able to cast something to any in typescript and get all the flexibility of javascript, but also have the rest of your code type checked if you want it. Statically typed languages usually don't let you mix-and-match static typing with duck typing.
Haha, I thought somehow Weird Al had made it to top.
Here's the song for the curious https://m.youtube.com/watch?v=eOL2q8leiLw
From 1985.
Here's the song for the curious https://m.youtube.com/watch?v=eOL2q8leiLw
From 1985.
And the followup: https://www.youtube.com/watch?v=Fk15H6PjBis
A take off on the classic Huey Lewis and the News song. Yes I had the Sports album, and saw Al live ~two years ago. :-D
Ah, the Ill Advised Self Indulgent Vanity Tour- I went and saw him in Eugene, OR. It was a fantastic show. I'll never forget it. 10 year old me was freaking out that he was playing the oddballs like "Those were the days" and "Stuck in a closet with Vanna White". What a show!
Yes, he came up near our seat at one point. Was a great show also, but I missed the MJ-related numbers. Only later I figured out why.
Why was that? I have not heard.
Edit: Answered my own question via Google: https://consequenceofsound.net/2019/06/weird-al-yankovic-mic...
Edit: Answered my own question via Google: https://consequenceofsound.net/2019/06/weird-al-yankovic-mic...
In 1985 I pretty much though Weird Al was the king. Crazy that he's still kicking it!
(For those living in the python + typing world, I'd definitely recommend trying Microsoft's pyright as an alternative to mypy. In my experience, it is substantially faster and produces higher quality error messages. It does not support mypy plugins, so -- for instance -- Django ORM dynamism isn't well captured. But I've found that the Django plugin for mypy is still pretty young; when used on a big codebase, it both slows down type checking considerably and regularly produces spurious errors.
Random: as someone who makes heavy use of both Python and Typescript, it pleases me to no end to see a great parser and ast for python implemented in typescript.)
Random: as someone who makes heavy use of both Python and Typescript, it pleases me to no end to see a great parser and ast for python implemented in typescript.)
TypeScript will eat the world one day, and honestly I don’t mind at all. Such a surprisingly powerful system, while being simple enough to throw juniors in the deep end with.
My personal website is being rebuilt on Deno at the moment! Writing type level operators is a lot of fun haha
My personal website is being rebuilt on Deno at the moment! Writing type level operators is a lot of fun haha
I've been using pyright for a couple of weeks because other LSP backends for Emacs weren't working well and only pyright seems to have any future. I am quite happy with its fast and effective type checking and it's not very hard to make it happy unlike mypy.
It's so bizarre to me - though I guess in-line with me finding a lot of decisions around Python 3 bizarre - that after years of Python being one of duck typing's biggest defenders and success stories, mypy went with nominal typing by default.
Both Go and TypeScript were right there showing you how effective it was.
Both Go and TypeScript were right there showing you how effective it was.
I heavily use and really appreciate mypy for what it is - but I also totally agree with you here. Duck typing _is_ structural typing, and it's pretty disappointing that this wasn't the focus of their efforts.
Yep. An absolute travesty, and makes working with mypy a pretty huge pain, even though the advantages of static typing still make it worth it. But man, when I switch back to Typescript, I remember how nice it could have been.
Mypy will only ever make it into a small fragment of Python code. Its existence is more likely an example of the those insistent on type checking crossing the picket line and bringing their last holdout features to the language than it is an acceptance that duck typing is not the way.
For the record, duck typing is the way. Mypy is a curiosity to me; I'm happy those who need it are getting that itch scratched; I'm happier it's boxed away from what I work with.
For the record, duck typing is the way. Mypy is a curiosity to me; I'm happy those who need it are getting that itch scratched; I'm happier it's boxed away from what I work with.
I thought like you at one point, but then jumped into a moderately-sized project that I was unfamiliar with. It took months to get an intuition for what the data was that was being passed around.
Inside a function you need modify, you are passed a foo. What can you do with it? Take the length? Add a number to it? You don't know unless you know the type, and then the only place that info exists is in your head. Why not write it down in a way that can be checked and therefore never get out of date, like comments or some weird reincarnation of Hungarian notation?
As a bonus: Editors will tell you when you are passing incorrect types to functions before you even run your tests. No more strings accidentally treated as lists!
I am now absolutely convinced that types are important, even in python.
Inside a function you need modify, you are passed a foo. What can you do with it? Take the length? Add a number to it? You don't know unless you know the type, and then the only place that info exists is in your head. Why not write it down in a way that can be checked and therefore never get out of date, like comments or some weird reincarnation of Hungarian notation?
As a bonus: Editors will tell you when you are passing incorrect types to functions before you even run your tests. No more strings accidentally treated as lists!
I am now absolutely convinced that types are important, even in python.
I remain unconvinced that the utility in the scenarios you describe outweighs the overhead in scenarios where it is irrelevant.
Even as nothing more than lightweight machine checked documentation, mypy is still a win. And just like you might write a few paragraphs of docs for one function and no docs at all for another, you can specify the hell out of one function's types and lean another completely unannotated. Mypy is an amazing middle ground because you don't need to move your whole codebase.
The overhead is as much or as little as you like.
The overhead is as much or as little as you like.
To each their own. I find almost all functions I write are really designed with one type in mind anyway.
One difference may be if you primarily work in your own projects or work with other people’s projects. And how skilled the other developers are. In my own projects I instinctively know the types, but not in other projects, and having the types documented allows me to make changes much more quickly.
One difference may be if you primarily work in your own projects or work with other people’s projects. And how skilled the other developers are. In my own projects I instinctively know the types, but not in other projects, and having the types documented allows me to make changes much more quickly.
> One difference may be if you primarily work in your own projects or work with other people’s projects.
Type-advocates throw this around all the time as if it's not the most patronizing thoughtpattern. Yes, I work on a good sized project with another 15 engineers using Python for the backend and angular for the frontend. If scale were going to reveal something dramatically different than whatever toy algorithms you might imagine I'm playing with...it would've happened by now.
Type-advocates throw this around all the time as if it's not the most patronizing thoughtpattern. Yes, I work on a good sized project with another 15 engineers using Python for the backend and angular for the frontend. If scale were going to reveal something dramatically different than whatever toy algorithms you might imagine I'm playing with...it would've happened by now.
I think you took this differently than what I intended, although that is partly my fault.
What I meant was more along the lines of “do you regularly jump into the deep end on new projects that were started/maintained by someone else?”
Doing that is what made me a believer. Jumping into my current project, taking over from some else, was a nightmare without types. I had no idea if something was a string, a dict, a dataclass, a custom class, etc, with only vague hints based on how it was being used. These obviously had “types”, and the functions were designed for only one set of types. I just couldn’t remember what.
“Your own” doesn’t necessarily mean small or toy. But more about consistently working in familiar code bases where you are already familiar with what types are flowing around.
I think of these kinds of types as enforced documentation. It’s not for you (necessarily), it’s for other people.
What I meant was more along the lines of “do you regularly jump into the deep end on new projects that were started/maintained by someone else?”
Doing that is what made me a believer. Jumping into my current project, taking over from some else, was a nightmare without types. I had no idea if something was a string, a dict, a dataclass, a custom class, etc, with only vague hints based on how it was being used. These obviously had “types”, and the functions were designed for only one set of types. I just couldn’t remember what.
“Your own” doesn’t necessarily mean small or toy. But more about consistently working in familiar code bases where you are already familiar with what types are flowing around.
I think of these kinds of types as enforced documentation. It’s not for you (necessarily), it’s for other people.
Please look at languages with good structural typing (e.g. Go - not so powerful but lots of syntactic convenience - and TypeScript - lots of power but you pay in compilation time, albeit not worse than mypy).
If mypy is doomed to remain marginal it is because it picked a poor type system for the language it tries to support, not because stricter typing generally is the wrong choice for that language.
If mypy is doomed to remain marginal it is because it picked a poor type system for the language it tries to support, not because stricter typing generally is the wrong choice for that language.
> Mypy will only ever make it into a small fragment of Python code.
I don't think this has to be the case. If it got serious investment and the Python community was more open to making ergonomic syntax improvements so type annotations could be natural, we could see improvement. Similarly, I think it would need to figure out a saner solution to finding/loading type annotations, because the current set up and error messaging are immensely painful. I also don't know if Mypy will ever be sufficiently performant so long as it's written in Python (I think people really underestimate the difference between instant feedback and a delay of several seconds). Having good editor support would also drive adoption.
If Mypy were able to improve on all of these distinct problem areas, I think more people would opt into type annotations naturally, but it certainly feels like Mypy is being treated as a thing to pacify the people who whine about static typing (of course I don't think that's the real intention, only how it comes across).
I don't think this has to be the case. If it got serious investment and the Python community was more open to making ergonomic syntax improvements so type annotations could be natural, we could see improvement. Similarly, I think it would need to figure out a saner solution to finding/loading type annotations, because the current set up and error messaging are immensely painful. I also don't know if Mypy will ever be sufficiently performant so long as it's written in Python (I think people really underestimate the difference between instant feedback and a delay of several seconds). Having good editor support would also drive adoption.
If Mypy were able to improve on all of these distinct problem areas, I think more people would opt into type annotations naturally, but it certainly feels like Mypy is being treated as a thing to pacify the people who whine about static typing (of course I don't think that's the real intention, only how it comes across).
No, no, no. Strong static typing is an unmitigated win. A stitch in time saves nine: with static typing you can check for type constraint violations at compile time that would otherwise throw runtime errors, eliminating a large class of bugs before your program even runs.
What's more, with static typing your IDE can provide you with more helpful guidance by immediately showing you the valid method names associated with an object, type-checking their parameters as you use them, and suggesting possible parameters of the required types from the variables currently in scope.
With dynamic/duck typing, you give all that up -- for no clear advantage.
Use static typing in a project of significant size. Always.
What's more, with static typing your IDE can provide you with more helpful guidance by immediately showing you the valid method names associated with an object, type-checking their parameters as you use them, and suggesting possible parameters of the required types from the variables currently in scope.
With dynamic/duck typing, you give all that up -- for no clear advantage.
Use static typing in a project of significant size. Always.
> eliminating a large class of bugs before your program even runs
And introducing complexity you have to resolve before your program even runs too! Complexity you have to solve. Static typing systems require you to do more design upfront. And more importantly, in your head! The most you can do is run your compiler and it'll tell you if you got it right. If not, just try again!
Dynamic systems will encourage you to code flexibility from the beginning. You may have to run your system to expose a class bugs...but so? I was going to run it anyway.
All these conversations are just about moving work around and when it gets done. Most problems are solved equally by a dynamic or static type system. However, static type systems inherently require more boilerplate. At the end of the day, you just rarely need it to do anything of importance.
And introducing complexity you have to resolve before your program even runs too! Complexity you have to solve. Static typing systems require you to do more design upfront. And more importantly, in your head! The most you can do is run your compiler and it'll tell you if you got it right. If not, just try again!
Dynamic systems will encourage you to code flexibility from the beginning. You may have to run your system to expose a class bugs...but so? I was going to run it anyway.
All these conversations are just about moving work around and when it gets done. Most problems are solved equally by a dynamic or static type system. However, static type systems inherently require more boilerplate. At the end of the day, you just rarely need it to do anything of importance.
Refactoring is basically impossible to do confidently in undisciplined dynamic codebases because you have not documented your types in the first place. Every codebase you have ever worked with is typed. You can either decide to ignore it and push the burden of memorizing what they are onto your developers, slowing your iteration rate, or you can spend the up front time to formalize it and let a program tell you what bugs you have. Those bugs already existed, it's not the type system's fault for pointing them out.
The formalization step also allows you to encode business logic, where it would not otherwise exist. Suppose you have a system with documents. Every document has a length property...today. So in an untyped codebase, you've been doing just fine assuming it exists, and "just running the code" has found no issues. But technically, it doesn't always exist, and this fact isn't encoded anywhere. It breaks. If you had just defined a document type, then you could simply update the length type and let mypy tell you literally all the places where the bug would arise. Or, it would have allowed you to avoid making that assumption in the first place, rather than trying to make sure every developer memorizes this useless fact and writing tests every time the length property is touched.
The formalization step also allows you to encode business logic, where it would not otherwise exist. Suppose you have a system with documents. Every document has a length property...today. So in an untyped codebase, you've been doing just fine assuming it exists, and "just running the code" has found no issues. But technically, it doesn't always exist, and this fact isn't encoded anywhere. It breaks. If you had just defined a document type, then you could simply update the length type and let mypy tell you literally all the places where the bug would arise. Or, it would have allowed you to avoid making that assumption in the first place, rather than trying to make sure every developer memorizes this useless fact and writing tests every time the length property is touched.
> Every codebase you have ever worked with is typed.
You've baked your conclusion into your argument. But by this logic, sure, everything is typed. I'll buy it. A static type system just makes you explicitly type it out for every single thing and again for every single thing that might operate on that first thing. It's pedantic, filled with boilerplate, and plenty of promises about problems the system solved that were just invisible before. Hmm...those might be the marks of snake oil, now aren't they?
Consider a iron foundry. Are there type checks on the iron? No, the entire process is duck-typing. Why? Because that's how reality works. You can specially shape your iron ingots to key into your foundry. What's that? Duck typing. The firing profile will follow certain characteristics. Did it get those characteristics from the ingot? No, it duck types it and starts firing it as if its iron. If it isn't iron, the process will throw an exception. And that part has all sorts of safety checks around it. The most you can do would be some type of chemical check on the ingot before starting. Almost like a typecheck before proceeding in a method.
Aristotelian vs. Platonic approaches. The problem with the Platonic approach is that it requires you to be an oracle. Static typing is like trying to encode logic at the molecular level. This atom WILL NOT bond this atom. I mean, sure, it can work...but there's a lot of atoms and a lot of interactions. Don't define what you don't need.
You've baked your conclusion into your argument. But by this logic, sure, everything is typed. I'll buy it. A static type system just makes you explicitly type it out for every single thing and again for every single thing that might operate on that first thing. It's pedantic, filled with boilerplate, and plenty of promises about problems the system solved that were just invisible before. Hmm...those might be the marks of snake oil, now aren't they?
Consider a iron foundry. Are there type checks on the iron? No, the entire process is duck-typing. Why? Because that's how reality works. You can specially shape your iron ingots to key into your foundry. What's that? Duck typing. The firing profile will follow certain characteristics. Did it get those characteristics from the ingot? No, it duck types it and starts firing it as if its iron. If it isn't iron, the process will throw an exception. And that part has all sorts of safety checks around it. The most you can do would be some type of chemical check on the ingot before starting. Almost like a typecheck before proceeding in a method.
Aristotelian vs. Platonic approaches. The problem with the Platonic approach is that it requires you to be an oracle. Static typing is like trying to encode logic at the molecular level. This atom WILL NOT bond this atom. I mean, sure, it can work...but there's a lot of atoms and a lot of interactions. Don't define what you don't need.
> Are there type checks on the iron?
Yes there are, it's called physics haha. The iron foundry analogy is not great imo. Programming is more akin to building an engine, but also building the tools that are used to make that engine. Because of the physical nature of these tools, you get a simple static type system by default: you cannot put a screw into a hole not meant for the screw with that thread. Now imagine you didn't have this guarantee. Any tool could technically be put anywhere, and you wouldn't know you had it wrong until you ran it and it killed you. Or even worse, it seems to run fine until you go over a bump, then it kills you. That's duck typing without a type system.
And god forbid you need to add more engineers to your engine project. Nothing is documented, and they have to infer the properties of the tools based on how they are used today.
Yes there are, it's called physics haha. The iron foundry analogy is not great imo. Programming is more akin to building an engine, but also building the tools that are used to make that engine. Because of the physical nature of these tools, you get a simple static type system by default: you cannot put a screw into a hole not meant for the screw with that thread. Now imagine you didn't have this guarantee. Any tool could technically be put anywhere, and you wouldn't know you had it wrong until you ran it and it killed you. Or even worse, it seems to run fine until you go over a bump, then it kills you. That's duck typing without a type system.
And god forbid you need to add more engineers to your engine project. Nothing is documented, and they have to infer the properties of the tools based on how they are used today.
> it seems to run fine until you go over a bump, then it kills you
That's half of duck-typing. You're missing the exception response where safety is enforced. A static type system fails with the wrong type. A duck-typed system responds with something as sensible as it can reason with what it was handed. Must often, that's a hard failure because most code is brittle and doesn't need the robustness (i.e. don't build what you aren't going to need).
That's half of duck-typing. You're missing the exception response where safety is enforced. A static type system fails with the wrong type. A duck-typed system responds with something as sensible as it can reason with what it was handed. Must often, that's a hard failure because most code is brittle and doesn't need the robustness (i.e. don't build what you aren't going to need).
This made me curious, so I’m taking a poll.
https://twitter.com/glyph/status/1372640855807250432?s=21
https://twitter.com/glyph/status/1372640855807250432?s=21
The results of the poll are in, and 59.6% of the 699 respondents said that they're either using it already or will be using it within the year. This isn't necessarily a representative sample, my followers are probably on the more advanced end of the Python spectrum, but 700 people (including myself) isn't nothing, either. I think it's likely that type annotations will be available for the majority of Python libraries.
I think this isn't quite what I want, because implicit structural typing has a single shared namespace of methods. If the Ducky protocol says "implementers should have .quack()" and I define FringeScientist().quack(), Mypy will think it's Ducky but it's just a naming coincidence. I want to say which protocols I'm implementing. Mypy can't tell whether my Foo.quack() is supposed to be an implementation of FringeScientist.quack() or Ducky.quack().
This is only a problem if the `FringeScientist.quack` signature is identical to Ducky.quack:
$ cat test.py
from dataclasses import dataclass
from typing import Protocol
class Ducky(Protocol):
def quack(self) -> None:
"Quack."
class FringeScientist:
def quack(self) -> bool:
return True
@dataclass
class Duck:
quiet: bool = False
def quack(self) -> None:
print("Quack." if self.quiet else "QUACK!")
def duck_war(aggressor: Ducky, defender: Ducky) -> None:
aggressor.quack()
defender.quack()
print("The only winning move is not to play.")
duck = Duck()
asshat = FringeScientist()
duck_war(duck, asshat)
$ mypy test.py
test.py:32: error: Argument 2 to "duck_war" has incompatible type "FringeScientist"; expected "Ducky" [arg-type]
test.py:32: note: Following member(s) of "FringeScientist" have conflicts:
test.py:32: note: Expected:
test.py:32: note: def quack(self) -> None
test.py:32: note: Got:
test.py:32: note: def quack(self) -> boolIn practice this happens an order of magnitude less than "I want to handle two things that match the same protocol that I didn't write and can't extend" and "I need to pass something that would work fine to a poorly-typed library" combined.
That's before we even get into the philosophical issue of, who are you, the lowly library implementor, to say those aren't something appropriate to abstract over in my application?
That's before we even get into the philosophical issue of, who are you, the lowly library implementor, to say those aren't something appropriate to abstract over in my application?
How can two objects present exactly the same API without either one knowingly emulating the other? Examples always seem to use one method taking no args, which don’t seem realistic.
What's missing in this example is stub typing of the output values.
Scientist.quack really shouldn't be returning plain strings, and neither should ducky.quack if the data in them is contextually different.
Things should only become basic types at the last possible moment.
Scientist.quack really shouldn't be returning plain strings, and neither should ducky.quack if the data in them is contextually different.
Things should only become basic types at the last possible moment.
Which language ecosystems operate like that in practice?
Go has no subtyping and only supports polymorphism via structural types (and some weak support for untyped constant values). It also has very convenient syntax to define newtypes from primitives (or non-primitives, but this is less commonly used).
Before Android supported Optional, there were dozens of independent reimplementations of the API. All exactly identical, but all in different namespaces and therefore not compatible.
And the official one obviously couldn't be used.
And the official one obviously couldn't be used.
A common example I ran into is machine learning methods and data transformations such as scaling have have `fit` and `transform` methods, and I'd like to write code that can take multiple such objects and compose them in a pipeline, along with some custom logic. Structural typing makes that easy to typecheck, nominal typing makes it difficult because those two classes don't have a parent class that contains those methods.
Extending this, if the class has a `partial_fit` method, sometimes I want to use that, and fallback to `fit` when it doesn't.
Extending this, if the class has a `partial_fit` method, sometimes I want to use that, and fallback to `fit` when it doesn't.
Nominal typing just means someone has to declare that LinearClassifier implements Classifier. That could be done by the LinearClassifier maintainer or by the user.
What languages let the user do this?
Python's zope.interface can do it. Julia's still figuring out their trait system, but some implementations allow that. I don't know Clojure but I wouldn't be surprised to hear if their Spec allows it.
I don't think either zope.interface or spec is really thought of as a nominal typing system? Certainly not spec. zope.interface of course can late-bind classes to interfaces but only because it's actually a runtime thing and Python late-binds everything.
One thing I did come across once before is when two APIs do something similar such as parsing and provide the same name. A couple of years ago I remember I was working with an API in Go that the function Scan which was `fn(interface{}) -> Error`, which was used to decode SQL values. I pulled in another library which provided that function but in a separate context - text parsing.
The interfaces matched, the compiler accepted it, but the application blew up at runtime.
The interfaces matched, the compiler accepted it, but the application blew up at runtime.
Consider an immutable list and a read-only view of a mutable list (that another thread may mutate concurrently). They offer exactly the same API, but it's a major mistake to pass one where the other was expected.
> Examples always seem to use one method taking no args, which don’t seem realistic.
This is one of the reasons it is so rare to actually have confusable types in a real program.
This is one of the reasons it is so rare to actually have confusable types in a real program.
Well, that's duck typing - looks like a duck, quacks like a duck, it's a Duck[y]. And while it's obviously hard to extrapolate simple examples like `duck_war` to real-world use-cases, I think signatures being different between methods of the same name on different protocols is generally enough to differentiate between them.
That's the point of protocols. It doesn't matter if it's a duck or not, what matters is that it can quack. Like in case of iterable, it doesn't matter what it is, what matters is that you can iterate over it. You can as well rely inheritance if you want.
Don't Golang interfaces work this way as well? (Not that I'm a fan).
This is why Rust makes you explicitly implement a trait for a given type, even if the type doesn't require any new methods to be implemented
It's a wee bit amusing to see the major icons of duck typing, python and javascript, slowly converge on the truth: static typing is a powerful and useful tool for mature projects, that should not be dismissed.
Looks like mypy has discovered interfaces. Lovely.
Looks like mypy has discovered interfaces. Lovely.
Seems more like a convergence towards gradual typing than to static typing.
I think you're spot on here. It turns out that a lot of software development is made easier with static types - but also that some of it is _not_. And having access to both without fighting the language can be very powerful.
I use static Python types a lot at the application layer, where things have no need to be particularly extensible and having strict definitions for how things are put together brings clarity. But at lower levels of the system, Python's duck typing really shines because I can solve ten slight variations on the same problem with the same code.
The same could be said of Typescript, or Clojure+spec, or really any other language where dynamic types are given to you for free (unlike, say, reflection in Java) but types can be added wherever you find them useful.
I use static Python types a lot at the application layer, where things have no need to be particularly extensible and having strict definitions for how things are put together brings clarity. But at lower levels of the system, Python's duck typing really shines because I can solve ten slight variations on the same problem with the same code.
The same could be said of Typescript, or Clojure+spec, or really any other language where dynamic types are given to you for free (unlike, say, reflection in Java) but types can be added wherever you find them useful.
Type elision or similar is available in many statically typed languages, and has been for a long while.
The difference between "Interfaces" as I have seen them used most of the time, and "Protocols" as mypy uses them, is that usually an Interface is generally part of the inheritance hierarchy.
A Protocol is just the same structural subtyping (duck typing) Python has always preferred.
A Protocol is just the same structural subtyping (duck typing) Python has always preferred.
[deleted]
These protocols match the old Indiana `concepts` pretty precisely; they appear to be more like Haskell type-classes than interfaces -- they're retroactive, not inherited.
Just speaking for myself here, I didn’t realize the power of static typing for years and years because the main languages I used with “static types” were C-style; i.e. type checking without null checking. I was vaguely aware of “better” systems but since I had such a wealth of experience with compile-time type-checking catching ~zero bugs, it didn’t seem worthwhile to investigate, and my toy projects didn’t make it clear what I was missing. Having used Mypy in a large practical situation for a couple of years, one of the MAIN advantages of it is the None checking! I would get probably 60% of the utility out of a type checker that could only check for None.
Having had this experience now it’s much easier to appreciate the practical benefits of systems like Rust and Haskell, where the type checker is doing real work.
Having had this experience now it’s much easier to appreciate the practical benefits of systems like Rust and Haskell, where the type checker is doing real work.
I could never get why Python added unchecked static typing. That seems so useless. If the compiler checks it, it can use it to optimize. Huge win for built-in types, like integer and float. CPython has to box everything. PyPy has to try to guess and use just-in-time recompilation when it guesses wrong.
For now, it improves documentation and static typechecking. But Mypyc is an annotated-python-to-C compiler that's in the works.
Type hints are super handy when using smart IDEs like PyCharm, which will list or highlight (most of) your type errors. This alone can greatly improve confidence in your program, especially for more complex cases.
Getting that cheaply while still having types unchecked is a nice compromise between improved functionality and not changing the language majorly.
Getting that cheaply while still having types unchecked is a nice compromise between improved functionality and not changing the language majorly.
Unchecked types by themselves are useful as they help my text editor to provide more accurate autocomplete suggestions.
Python didn't add static typing. It added arbitrary annotations of arguments and variables (admittedly, with the use case of type checking in mind). Mypy uses that for type annotations and checking. Here are the two main reasons I think it's useful:
* To the extent that it enables some static analysis on python code, it has often caught a number of errors I tend to make while writing code
* It serves as a light form of documentation that tends to stay more fresh because there's an automated way to check its consistency (run the type checker)
I don't expect that it will do as perfect a job as Rust or C++'s type system or even that it would be there for the same reasons. On the other hand, it's nice to just have more info about what the code is expected to do in the form of type annotations.
* To the extent that it enables some static analysis on python code, it has often caught a number of errors I tend to make while writing code
* It serves as a light form of documentation that tends to stay more fresh because there's an automated way to check its consistency (run the type checker)
I don't expect that it will do as perfect a job as Rust or C++'s type system or even that it would be there for the same reasons. On the other hand, it's nice to just have more info about what the code is expected to do in the form of type annotations.
> Python didn’t add static typing. It added arbitrary annotations of arguments and variables (admittedly, with the use case of type checking in mind). Mypy uses that for type annotations and checking.
Mypy started out as its own language with a Typescript-like relation to Python, evolved to being fully python compatible (using typing comments) after consultation with Guido (then-BDFL) and then python added annotations to support projects like mypy. It wasn’t just the that annotations were created and then mypy happened along to take advantage of them, supporting mypy specifically (without only supporting mypy) was a key motivation for annotations.
Mypy started out as its own language with a Typescript-like relation to Python, evolved to being fully python compatible (using typing comments) after consultation with Guido (then-BDFL) and then python added annotations to support projects like mypy. It wasn’t just the that annotations were created and then mypy happened along to take advantage of them, supporting mypy specifically (without only supporting mypy) was a key motivation for annotations.
The biggest problem with Python typing + mypy is that it's not complete: not everything can be type annotated strictly (as in, avoiding `Any`) and correctly (as in, matching actual types passing through the code and not just whatever shuts up mypy). One example is JSON-like `dict`s, since mypy doesn't support recursive types yet[1].
[1] https://github.com/python/mypy/issues/731
[1] https://github.com/python/mypy/issues/731
Where would you use `typing.Protocol` where you wouldn't use an abstract base class (or a bunch of them as mixins)? My understanding is that the latter is more "formal", whereas `Protocol` is better suited for gradual typing (i.e., moving a more "free-form" codebase into something MyPy won't complain about).
The only time I've ever used `Protocol` is to define a type that makes it explicit that I need an object to have a `__str__` implementation:
The only time I've ever used `Protocol` is to define a type that makes it explicit that I need an object to have a `__str__` implementation:
@runtime_checkable
class Stringable(Protocol):
""" Protocol type that supports the __str__ method """
@abstractmethod
def __str__(self) -> str:
""" Return the string representation of an object """
This distinction is the same as the distinction between structural and nominal typing, described at the end of the article.
Abstract base classes require everything to extend from a base-level object, and also inherit it's default implementations. This is a nominal (name-based) mechanism.
Protocol (structural-based) subtyping instead only declares what methods something has to have, without requiring it tie itself to either to a concrete parent class and its behaviours.
Edit: since you partially address this, you ask why you'd use structural typing rather than an abstract base class. I'll turn it around and ask why you would use an abstract base class, which requires modifying the children accordingly, when you could instead use a structural type. The usual preference comes down to whether you want to be explicit about implementing it or not, and whether you want to pull in default behaviour.
Abstract base classes require everything to extend from a base-level object, and also inherit it's default implementations. This is a nominal (name-based) mechanism.
Protocol (structural-based) subtyping instead only declares what methods something has to have, without requiring it tie itself to either to a concrete parent class and its behaviours.
Edit: since you partially address this, you ask why you'd use structural typing rather than an abstract base class. I'll turn it around and ask why you would use an abstract base class, which requires modifying the children accordingly, when you could instead use a structural type. The usual preference comes down to whether you want to be explicit about implementing it or not, and whether you want to pull in default behaviour.
> The usual preference comes down to whether you want to be explicit about implementing it or not, and whether you want to pull in default behaviour.
Thanks :) I definitely prefer things to be explicit; regardless of it being a "Zen of Python" mantra. That said, I do see the value of `Protocol` for non-annotated/legacy code, in that it gives you a convenient mechanism for this. I would be worried about it becoming a crutch, if overused, but definitely better than no annotations at all!
(I don't understand your point regarding an ABC's default implementation. Isn't the point of ABCs that they're not base classes, but interfaces that define the methods required without any implementation?)
Thanks :) I definitely prefer things to be explicit; regardless of it being a "Zen of Python" mantra. That said, I do see the value of `Protocol` for non-annotated/legacy code, in that it gives you a convenient mechanism for this. I would be worried about it becoming a crutch, if overused, but definitely better than no annotations at all!
(I don't understand your point regarding an ABC's default implementation. Isn't the point of ABCs that they're not base classes, but interfaces that define the methods required without any implementation?)
> Abstract base classes require everything to extend from a base-level object, and also inherit it's default implementations. This is a nominal (name-based) mechanism.
The very definition of an abstract base class precludes the existence of (meaningful) implementations. If the base class has implementations for the methods expected to be overridden (beyond no-ops or throwing some exception around the lack of an implementation), then it is definitionally not abstract.
That is: a protocol and an abstract class achieve the same thing: they declare the existence of methods, and defer implementation of those methods to implementations / child classes (respectively).
Interfaces are great, but if you have multiple inheritance as your hammer (as you do in Python), then it's perfectly reasonable for abstract class inheritance to be your nail.
The very definition of an abstract base class precludes the existence of (meaningful) implementations. If the base class has implementations for the methods expected to be overridden (beyond no-ops or throwing some exception around the lack of an implementation), then it is definitionally not abstract.
That is: a protocol and an abstract class achieve the same thing: they declare the existence of methods, and defer implementation of those methods to implementations / child classes (respectively).
Interfaces are great, but if you have multiple inheritance as your hammer (as you do in Python), then it's perfectly reasonable for abstract class inheritance to be your nail.
(meaningful) isn't as much a get-of-out-jail clause here as you suggest, if I understand correctly. Requiring implementations at all, even if they are broken, moves the time when you find out about a missing definition to runtime, rather than being statically determined at typecheck time.
If the not-overriden method isn't commonly called, it's possible for the non-meaningful / raise an exception version to make it to production. Not so if the error happens at typecheck time.
If the not-overriden method isn't commonly called, it's possible for the non-meaningful / raise an exception version to make it to production. Not so if the error happens at typecheck time.
Right, but that's a problem more fundamental to Python and applicable far beyond the "abstract" classes hacked onto it. That is: if the methods were missing entirely, you'd have the same problem, since Python (without the help of external tools) doesn't make any attempt to validate these things at (bytecode) compile time.
That is true. However, other linters (e.g., pylint) will check a class satisfies its interface. IIRC, the runtime error for non-implemented abstract methods happens at instantiation time, rather than when said missing method call is attempted.
> The very definition of an abstract base class precludes the existence of (meaningful) implementations.
No, it doesn't. It precludes meaningful implementation of the complete interface, of course, but it doesn't preclude meaningful implementation for some methods, which depend on the other methods do which meaningful implementations are not provided.
No, it doesn't. It precludes meaningful implementation of the complete interface, of course, but it doesn't preclude meaningful implementation for some methods, which depend on the other methods do which meaningful implementations are not provided.
Right, but those methods which are implemented are not in the set of methods intended for descendant classes to override. If they are in that set, at all, then the base class is by definition not abstract.
The fundamental issue is that Python doesn't really have a way to enforce this, so there's ultimately no such thing as an abstract class in Python - there are only classes that act like they're abstract (by replacing all implementations of meant-to-be-overridden methods with no-ops or thrown exceptions).
The fundamental issue is that Python doesn't really have a way to enforce this, so there's ultimately no such thing as an abstract class in Python - there are only classes that act like they're abstract (by replacing all implementations of meant-to-be-overridden methods with no-ops or thrown exceptions).
> The fundamental issue is that Python doesn't really have a way to enforce this
When (as to make this even worth discussing, one must) one includes the typecheckers (which are technically external) in “Python”, that's not true; both mypy and pyright enforce that abstract classes (either those explicitly declared as abcs, or derived classes with any non-overridden declared-as-abstract methods) cannot be instantiated.
When (as to make this even worth discussing, one must) one includes the typecheckers (which are technically external) in “Python”, that's not true; both mypy and pyright enforce that abstract classes (either those explicitly declared as abcs, or derived classes with any non-overridden declared-as-abstract methods) cannot be instantiated.
> Abstract base classes require everything to extend from a base-level object, and also inherit it's default implementations
The general OOP concept of an “abstract base class” might, the particular Python implementation does not, because, as well as classic inheritance, it supports the concept of “virtual subclasses” that are registered with, but do not actually inherit from (and thus do not use the implementation of), an abc.
It shares with structural subtyping that it only defines a mandated interface, but with nominal subtyping that it requires an explicit statement of intent to conform to the interface before an object is considered to conform.
The general OOP concept of an “abstract base class” might, the particular Python implementation does not, because, as well as classic inheritance, it supports the concept of “virtual subclasses” that are registered with, but do not actually inherit from (and thus do not use the implementation of), an abc.
It shares with structural subtyping that it only defines a mandated interface, but with nominal subtyping that it requires an explicit statement of intent to conform to the interface before an object is considered to conform.
> With Mypy, you get all the benefits of high-level dynamic typing for rapid experimentation, and all the benefits of rigorous type checking...
Except that most typed languages are far faster than python. Retrofitting existing python projects with types makes sense for reliability, but starting a new project with python and types makes less sense, imo.
Except that most typed languages are far faster than python. Retrofitting existing python projects with types makes sense for reliability, but starting a new project with python and types makes less sense, imo.
Using that logic, it never made sense to start any project with Python, correct?
People use Python for its ease and speed of development, and extensive ecosystem. Things that need to be fast can have the appropriate effort expended to execute as fast as necessary.
People use Python for its ease and speed of development, and extensive ecosystem. Things that need to be fast can have the appropriate effort expended to execute as fast as necessary.
> People use Python for its ease and speed of development, and extensive ecosystem.
Of these, I think only the ecosystem stands out anymore. For work purposes I wouldn't be able to move away from Python anytime soon, but if I could pick any language for new code it wouldn't be Python. If I had to pick the closest competitor, Julia seems like a great Python-like language in terms of speed and ease of development with much better typing and performance, but with a less extensive ecosystem. Nim might also fit that bill. If ecosystem was important, Go probably fills a similar space without being quite as steep of a learning curve as some of the other languages.
Of these, I think only the ecosystem stands out anymore. For work purposes I wouldn't be able to move away from Python anytime soon, but if I could pick any language for new code it wouldn't be Python. If I had to pick the closest competitor, Julia seems like a great Python-like language in terms of speed and ease of development with much better typing and performance, but with a less extensive ecosystem. Nim might also fit that bill. If ecosystem was important, Go probably fills a similar space without being quite as steep of a learning curve as some of the other languages.
> it never made sense to start any project with Python, correct?
Well as a Perl fan I certainly don't disagree with that conclusion ;)
But yeah, you're definitely right that there's immense value to being able to quickly prototype.
Well as a Perl fan I certainly don't disagree with that conclusion ;)
But yeah, you're definitely right that there's immense value to being able to quickly prototype.
Cython is an option when a (potential) 100x speedup is required. Choosing the right algorithm up front, thru the rapid experimentation Python gives, can sometimes make it faster than a real world implementation in a static language, that didn't get enough time to experiment.
The post isn't about performance, and is aimed at people using Python for whatever reason, so, sure, retrofit away.
That said, if you care about the performance improvements that typing can give you with Mypy, you might want to look here:
https://github.com/mypyc/mypyc-benchmark-results/blob/master...
It won't be going toe to toe with Rust any time soon, but a 4x to 17x speedup is nothing to sneeze at.
That said, if you care about the performance improvements that typing can give you with Mypy, you might want to look here:
https://github.com/mypyc/mypyc-benchmark-results/blob/master...
It won't be going toe to toe with Rust any time soon, but a 4x to 17x speedup is nothing to sneeze at.
That's cool if there's any indication that this could actually be used in production. Namely, what is the compatibility with the ecosystem. The history of Python could be aptly characterized as a series of big problems -> tools promising miraculous solutions -> tools utterly failing to deliver on those solutions because they failed to consider some important part of the ecosystem. We need more than a benchmark graph to suggest that this is a panacea to Python's longstanding performance problems.
Mypy itself uses this extensively; it was developed because typechecking itself was getting too slow. So it’s seen some practical real-world use. It’s not a panacea (heck, it’s barely documented) but it’s an appealing option if you have some clear hotspot in Python that you need to optimize.
There is a very similar tool called Cython that is used by scikitlearn. Coincidentally Cython also solves a lot of python 2/3 problems since it compiles for both.
Also, Mypy has a thousand paper cuts compared to other type systems. Pretty sure we still can't model JSON in Mypy (e.g., JSON = Union[List['JSON'], Dict[str, 'JSON'], None, bool, int, float, str] or whatever) because Mypy doesn't support recursive types. Moreover, since it's largely shoehorned into existing Python syntax, it's very awkward. E.g., I'm still not sure what the scoping rules are for TypeVar, describing a callback that takes kwargs involves defining a protocol class with a `__call__()` method that takes your function's kwargs, etc. Further, getting the type definitions to load properly is incredibly painful, the error message directs you to a page with a few things you might try to debug further, but never has my issue been among them. Also, like everything in the Python world, Mypy is slow, and you really want your type checks to be instant.
Most of these things can be improved upon, but progress feels slow and there are so many other serious issues in the Python ecosystem (performance, package management, etc) that I've given up (after 15 years of daily use). It makes me sad, but Go and Rust seem to offer better tradeoffs for the things that I care about.
Most of these things can be improved upon, but progress feels slow and there are so many other serious issues in the Python ecosystem (performance, package management, etc) that I've given up (after 15 years of daily use). It makes me sad, but Go and Rust seem to offer better tradeoffs for the things that I care about.
I start new projects with python and mypy. If you're convinced that there are some problems that better be solved in python, you can easily see that once you have a project in python, mypy just adds more safety to it. So there is nothing wrong with python+mypy for new projects.
Is this not literally the distinction between an interface and an implementation?
Like, we all know about the expression problem and how interfaces are hard to modify after the fact...
But are test mocks and stubs not literally the prime use case for interfaces at the syntax level?
Like, we all know about the expression problem and how interfaces are hard to modify after the fact...
But are test mocks and stubs not literally the prime use case for interfaces at the syntax level?
> Is this not literally the distinction between an interface and an implementation?
No. e.g. Java interfaces are just as nominal as classes.
> But are test mocks and stubs not literally the prime use case for interfaces at the syntax level?
Also no. If your language has structural interfaces it is often preferred to inheritance as a tool for polymorphism. Even in C++ where templates are not the easiest tool to wield it's often still preferable to inheritance for many reasons.
No. e.g. Java interfaces are just as nominal as classes.
> But are test mocks and stubs not literally the prime use case for interfaces at the syntax level?
Also no. If your language has structural interfaces it is often preferred to inheritance as a tool for polymorphism. Even in C++ where templates are not the easiest tool to wield it's often still preferable to inheritance for many reasons.
What? No. The author has a Duck, which has a quack() method. The quack() method is, presumably, identified by its name, argument types, and return type. Even in a language with purely nominal typing, you would make a Quacker interface, which is the set of things that quack() -> None, and then have both Duck and all of its test stubs be Quackers.
This is very literally how people who write in languages without structural types do test stubs (and often dependency injection, and other mechanisms that need to select from among several implementations of something).
This is very literally how people who write in languages without structural types do test stubs (and often dependency injection, and other mechanisms that need to select from among several implementations of something).
Alright, I guess I don't know what you mean by "prime use case" then. Yes, you can use interfaces to make test doubles. No, that's not the historically motivating factor behind their inclusion in languages.
This is one of those things that I wish type systems could just do for us. The pattern of every single class in Java being an an interface with an InterfaceImpl is exhausting.
Having to fight the type system just for testing seems silly. Just have every class's public bits define an interface and the class's name in code always refer to the interface.
Then testers can just implement ClassName with no fanfare and you stop having to write "test-friendly" classes.
Having to fight the type system just for testing seems silly. Just have every class's public bits define an interface and the class's name in code always refer to the interface.
Then testers can just implement ClassName with no fanfare and you stop having to write "test-friendly" classes.
My biggest pain as far as Python3 typing / mypy goes has been interfacing with major libraries and trying to add types in a way that isn't just me tossing `Any` everywhere.
If memory serves, Twisted hasn't been too bad. SqlAlchemy is pretty well a nightmare though. I think I saw that the latest release has type stubs. That will be nice. PyQt is also less than ideal...
If memory serves, Twisted hasn't been too bad. SqlAlchemy is pretty well a nightmare though. I think I saw that the latest release has type stubs. That will be nice. PyQt is also less than ideal...
This title made me laugh because as a kid I happily sang along with Huey in the 80's about wanting a new... truck. Didn't even stop to wonder how a truck would make one have all those feelings....
We are slowly getting some bits of static analysis. Perhaps another 20 years, and Python users will invent compilers.
> Perhaps another 20 years, and Python users will invent compilers.
Python’s main implementation has a bytecode compiler (that’s where pyc/pyo files come from.)
Mypyc compiles type-annotated python code to native-code python extensions.
Cython is a native-code compiler for a superset of python.
Both Mypyc and Cython are used for widely-used code.
Python’s main implementation has a bytecode compiler (that’s where pyc/pyo files come from.)
Mypyc compiles type-annotated python code to native-code python extensions.
Cython is a native-code compiler for a superset of python.
Both Mypyc and Cython are used for widely-used code.
Admittedly I've never done serious work with mypy (or typescript), so I'm approaching the value proposition of dynamic typing at face value rather than experience. However, it seems like the primary benefit of these languages was ease and flexibility, ableit at the cost of structure. Or said differently, adding mypy feels like trying to get out of a trade-off decision.
This situation reminds me of a talk Bryan Cantrill gave on platform core values, and as examples he gave his interpretation of the platform core values of languages like C, Awk, and Scala: https://youtu.be/2wZ1pCpJUIM?t=349
For me, platform core values that stick out for python would be Approachability, Simplicity, and Velocity. I understand the posited value mypy brings to the table, but it feels in contention with the original core values that made python appealing to begin with.