It is becoming difficult for me to be productive in Python(avi.im)
avi.im
It is becoming difficult for me to be productive in Python
https://avi.im/blag/2023/refactoring-python/
56 comments
I love typing. But python's type annotation is absolute garbage. The fact that you need to import types should tell you what a load of shit it is. All files now have type imports. Yuck.
In addition, the only way type annotations are effective is if you use an editor that can parse types. Yuck.
Lastly, it is far too easy to create complex types with questionable value all over your code. Any time a developer wants to unravel a pre-existing complex type, it feels like working with C++. It's utter garbage.
In addition, the only way type annotations are effective is if you use an editor that can parse types. Yuck.
Lastly, it is far too easy to create complex types with questionable value all over your code. Any time a developer wants to unravel a pre-existing complex type, it feels like working with C++. It's utter garbage.
>Lastly, it is far too easy to create complex types with questionable value all over your code. Any time a developer wants to unravel a pre-existing complex type, it feels like working with C++. It's utter garbage.
No. Having types is always better than not. If you have a complex functionality, it is going to have a complex type. But now you KNOW its complex and not have to unravel its complexity for the 5th time this month.
I had to write something like this recently:
```
pub SearchInfo {
```
This is complex, yes, but just by reading the type and names given to it:
1. `Arc`. So that means it can be safely shared across thread.
2. `HashMap` from `PathBuf` to `X`. So I am probably mapping file/folders to X.
3. `Vec<(usize, (usize, usize), String)>`. So the first `usize` denotes the line number. `(usize,usize)` denotes the start and end characters and the `String` is the whole line itself!
Just imagine unraveling this whole crap if its all simply a JavaScript `Object`.
No. Having types is always better than not. If you have a complex functionality, it is going to have a complex type. But now you KNOW its complex and not have to unravel its complexity for the 5th time this month.
I had to write something like this recently:
```
pub SearchInfo {
matches: Arc<HashMap<PathBuf, Vec<(usize, (usize, usize), String)>>>
```
This is complex, yes, but just by reading the type and names given to it:
1. `Arc`. So that means it can be safely shared across thread.
2. `HashMap` from `PathBuf` to `X`. So I am probably mapping file/folders to X.
3. `Vec<(usize, (usize, usize), String)>`. So the first `usize` denotes the line number. `(usize,usize)` denotes the start and end characters and the `String` is the whole line itself!
Just imagine unraveling this whole crap if its all simply a JavaScript `Object`.
I'd convert that last tuple to struct, unlike the rest of the type it's undecipherable without some kind of documentation.
It would be probably better if you haven't used naked usize, but wrapped it into a new type. When I find types get complex, I introduce type aliases or structs. That X in your signature likely deserves a name.
You can (and should) use mypy as part of your pipeline if you use type annotations, and that verifies that your types are correct without needing to use a special editor.
That doesn't work when many libraries don't use type annotations.
> You can (and should) use mypy as part of your pipeline if you use type annotations, and that verifies that your types are correct without needing to use a special editor.
As software engineers, if you are relying on CI to verify language features, you are going backwards in time.
As software engineers, if you are relying on CI to verify language features, you are going backwards in time.
It's not ci, it's part of your (local) build.
Not in my experience. If it's in your local build, it also needs to be in CI. Otherwise mistakes can easily slip through.
The point is it can be both places: locally during hacking and in CI to prevent accidentally pushing broken builds.
Isn't that what I just said?
Also mypy is super easy to use. You just run it on your repo root and it automatically checks everything. It's perfectly well-suited to CLI based dev workflows.
What type imports are you talking about?
https://peps.python.org/pep-0585/ has been implemented in Python 3.9.
https://peps.python.org/pep-0585/ has been implemented in Python 3.9.
Type annotations are not comparable to unit tests at all. Unit tests verifies correctness of the behavior, not just the types. E.g. if you have a mathematical function like min(), unit tests can verify it return the smallest of the inputs. Type annotations can at best verify that it returns a value of the same type as the input, but not if it is the smallest, largest or something random.
Type annotations are great for documentation and as a basic sanity test. They are especially useful with tools like code analysis or refactoring tools. But types are is no way a substitute for unit tests.
Type annotations are great for documentation and as a basic sanity test. They are especially useful with tools like code analysis or refactoring tools. But types are is no way a substitute for unit tests.
It tests at compile time that some function f takes A and returns B. That covers 99% of all possible input/output which you technically have to do in an untyped language. How do you make sure your function that takes an `any` doesn't crash and burn or silently sneak through and write null to the database?
> It tests at compile time that some function f takes A and returns B. That covers 99% of all possible input/output which you technically have to do in an untyped language.
Sorry, I don't understand what you are saying here. By A and B are you referring to the correct output given an input, or just to the expected data type? Because there is a big difference, as with the `min()` example.
Sorry, I don't understand what you are saying here. By A and B are you referring to the correct output given an input, or just to the expected data type? Because there is a big difference, as with the `min()` example.
A and B refers to the expected data type, yes.
So proving that `min(number, number)` returns a number is 99% as good as a unit test which shows that it returns the smallest number?
yes because the set of possible inputs for A -> C is basically infinite. But min(A,B)=C is only 2^64^2. Why shouldn't you test for string, object, object of object, object with a certain kind of shape, or Number class as opposed to number primitive? How can you know it will or won't work from a unit test perspective?
>inb4 just document it
that is what types are for
>inb4 just document it
that is what types are for
But the purpose of unit tests is to show that the code under test is correct, i.e. that min() return the smallest input, not just a random value of the same type. Static typing cannot do that, so it is in no way a substitute for unit tests.
Static typing is a useful tool, just not a substitute for any unit tests. The risk of static typing is when it becomes a goal in itself and you start thinking correctness means the program satisfies the type checker.
Static typing is a useful tool, just not a substitute for any unit tests. The risk of static typing is when it becomes a goal in itself and you start thinking correctness means the program satisfies the type checker.
One possible answer is that you make that a feature: the database can contain null, and it means something. In some cases you will trade one problem for another, but the economics of the tradeoff in overall productivity can be favorable.
That very much depends on the type system you use. With (for example) refinement types and dependent types you can prove your code correct with zero tests needed. Examples are F*, Idris, Agda, Coq etc.
Prove the code is correct per the design, it doesn’t eliminate the possibility of design errors which can still benefit from tests (or QuickCheck like tools) to ensure that what you design does what you think it does … a critical component given how rarely you see these languages used to actually build large applications in the wild.
The key difference is that with proven correct code you actually have a formal spec you can prove properties about. Without proven correct code you don’t even have a formal spec. Just a bunch of random tests (if you are lucky) that may or may not match your informal spec.
And the argument that if you can’t prove your design correct then there is no point proving your code correct is a strange one. That’s like saying that there is no point writing tests because you can’t prove your design correct or guarantee that all tests needed will be written. Ehhhh nope. I have written and generated more than 9000 tests for a very large scale C++ applications that are used by large corporations around the world. And I haven’t had a bug in production for 5+ years. However I of course can’t prove that those tests cover everything or that my design is correct. But that doesn’t mean it isn’t worth doing.
And the argument that if you can’t prove your design correct then there is no point proving your code correct is a strange one. That’s like saying that there is no point writing tests because you can’t prove your design correct or guarantee that all tests needed will be written. Ehhhh nope. I have written and generated more than 9000 tests for a very large scale C++ applications that are used by large corporations around the world. And I haven’t had a bug in production for 5+ years. However I of course can’t prove that those tests cover everything or that my design is correct. But that doesn’t mean it isn’t worth doing.
Yeah, I feel way too many people mistake "we can't cover 100% of everything" with "it's not worth tightening the bolts". Strange conflation but a very common one indeed.
All well-formed code is always a correct version of itself (Curry-Howard correspondence). It's just that sometimes you thought it was a correct version of something else and turn out to be wrong about that.
Since dependent type systems are just another kind of code, this is still true.
Since dependent type systems are just another kind of code, this is still true.
a * sin(x) + b * cos (-x)
a * sin(x) - b * cos (x)
In a lot of situations you have related values of exactly the same type: floats, strings, integers, Widget objects, which get mixed up.
A warning about variable shadowing or unused variable can be worth more than a type diagnostic. E.g. suppose
a * sin(x) - a * cos (x)
“lower down your business requirement”
I’m guessing you’ve never talked to someone requesting dev work on upwork
I’m guessing you’ve never talked to someone requesting dev work on upwork
> It's like discovering that bidet exists or something.
They're known and readily available in the US, but never really caught on, even during and after widespread toilet paper shortages in 2020.
Probably California would ban them anyway along with toilets that actually work.
They're known and readily available in the US, but never really caught on, even during and after widespread toilet paper shortages in 2020.
Probably California would ban them anyway along with toilets that actually work.
pick fp and haskell / ocaml/ racket/ common lisp
haskell
If that is the case, why not use Pythons's optional type annotations? Since the author didn't consider that, I suspect this is more about a love story than a tool evaluation. Rust my new love, I'm breaking up with Python.
> Years of coding in Go gave me this comfortable feeling: if it compiles, it works.
That is never a rational feeling.
> Type hints solve these issues to some extent, but bugs can still slip through.
But the link is to a tweet showing how a Python static analysis tool actually caught the bug.
> You could set up CI/CD pipeline enforcing 100% code coverage, but that will affect team productivity and surely piss off people.
If enforcing optional type hints would piss people off then, surely, using a statically typed language would also piss people off.
> Years of coding in Go gave me this comfortable feeling: if it compiles, it works.
That is never a rational feeling.
> Type hints solve these issues to some extent, but bugs can still slip through.
But the link is to a tweet showing how a Python static analysis tool actually caught the bug.
> You could set up CI/CD pipeline enforcing 100% code coverage, but that will affect team productivity and surely piss off people.
If enforcing optional type hints would piss people off then, surely, using a statically typed language would also piss people off.
You're 100% right, the new type annotations are really nice to. You can still be really pythonic and leverage some cool reflection capabilities with types. And on your point of forcing people to use types. I can't imagine many situations where you don't have a type in mind and can't use a higher order function to solve the problem. Which python does internally with __add__ __str__ and all the other functions included with classes.
As an example of stuff you can do with types. I was trying to teach a buddy how to program from a non CS background and I explored it a bit to make it as modular as possible so he could literally just change the structure and types and the code would all work for him. And it's really easy using some reflection
For example, to mimic a struct, you can use.
As an example of stuff you can do with types. I was trying to teach a buddy how to program from a non CS background and I explored it a bit to make it as modular as possible so he could literally just change the structure and types and the code would all work for him. And it's really easy using some reflection
For example, to mimic a struct, you can use.
@dataclass
class CSVStruct:
x: int = 0
y: int = 0
foo: int = 0
bar: int = 0
arg_list = []
# Generating argument list dynamically
for field in dataclasses.fields(self.CSVStruct):
arg_list.append(field.type(randint(0, 100)))
# Getting column names
row_names = {field.name: field.type for field in dataclasses.fields(self.CSVStruct)}I think comparing unit tests to a robust type system is a good way to think about typed languages. In my experience, unit tests most often catch NPEs, which a type system also does. The automated refactoring that can be done with a good type system is crazy.
Python has its place in a programmers tool belt, but I often ask questions to determine the language to use. How long lived will this code base be? How critical will be every ms of performance? How many LoC will the final product be? How many people will work inside this project? How dense and complex is the system to be built?
Python is incredible at short, utilitarian scripts that perform a single task that is not normally done inside of your main code base. It’s robust library collection lets you easily do things that other languages would need to be built from scratch. A short script to scrape twitter or analyze data can provide an enormous amount of value for time invested.
Python has its place in a programmers tool belt, but I often ask questions to determine the language to use. How long lived will this code base be? How critical will be every ms of performance? How many LoC will the final product be? How many people will work inside this project? How dense and complex is the system to be built?
Python is incredible at short, utilitarian scripts that perform a single task that is not normally done inside of your main code base. It’s robust library collection lets you easily do things that other languages would need to be built from scratch. A short script to scrape twitter or analyze data can provide an enormous amount of value for time invested.
It bothers me that people still treat python as a quick script hacking tool. You can build magnificent software with python.
I have written and am currently writing software in Python, and let me tell you: I would rather use any other statically typed language, such as Kotlin or F#.
In theory, yes. However, I have never actually seen this mythical magnificent software written in Python.
I've always found this game (in development) to be extremely impressive:
https://www.markrjohnsongames.com/games/ultima-ratio-regum/
https://www.markrjohnsongames.com/games/ultima-ratio-regum/
Magnificent software is software that makes money or ads value. With Python, technical debt has a higher interest rate, but the loan amount is greater. Right now, I've worked on a webapp that actually makes me money. It took my 6 weeks with really hacky python to get it running in production and serving requests. My last company is still 4 months on their iteration of a similar product because they wanted to build it the correct way with a typed language and better abstraction.
IMHO for greenfield startup projects/POCs, I much prefer python for making things fast and getting things out to market asap. Spaghetti gets me fed.
IMHO for greenfield startup projects/POCs, I much prefer python for making things fast and getting things out to market asap. Spaghetti gets me fed.
It sounds like you scoped the requirements tightly. Kept the code simple by minimizing LoC. likely you have no interest in writing and re-writing the product to maximize or increase profits overtime. You don’t have to work with anyone else to document or explain your spaghetti.
This is good for python. It does break down for bigger more complex projects at a certain point.
This is good for python. It does break down for bigger more complex projects at a certain point.
https://github.com/uktrade/tamato/
This is a Django based tool to manage the tax rates you pay on any thing you might trade with the UK.
It can handle all the changes to the tax from the founding of the EU, through past Brexit when the UK has its own tariff as well.
Running it locally, it may not be straightforward to get hold of the right data (you can download it, but I don't think it's a turn key thing).
This is a Django based tool to manage the tax rates you pay on any thing you might trade with the UK.
It can handle all the changes to the tax from the founding of the EU, through past Brexit when the UK has its own tariff as well.
Running it locally, it may not be straightforward to get hold of the right data (you can download it, but I don't think it's a turn key thing).
It's a bit of a hell to write anything in python that is more than a 100 lines or so and needs any amount of structure.
What's NPE?
I’m guessing you’ve never had to write Java
Null Pointer Exception
I used to like Python - not having strictly enforced types was nice. But when the code I wrote would involve multiple classes, different modules, etc. it became unwieldy. Since the switch to C#, I've gotten better at thinking about structure. It's very rare (read: never) that I run into a situation where I go, "DAMN, I wish I didn't have to define a type here - just put the var and let it work." I find that code is easier to reason about when I'm forced to think about how the data is going to flow.
You may end up enjoying f# quite a bit if you give it a try.
I actually failed an interview onetime because the interviewer (PHP guy) didn’t agree with me (C# guy, at the time) that having a statically typed language reduced the need for so many pedantic unit tests.
The (PHP guy) interviewer failed the interview with you.
My experience exactly using any untyped programming language. Fun for small throw away projects but not for serious large scale software that large corporations rely on.
[deleted]
[deleted]
> I used to think, ugh, I’d have to define types everywhere, make struct (or classes) in advance, and sometimes make wild guesses because those things were unknown or not finalised
A defined wild guess of something which is not finalised is still miles better than an undefined, implicit, amorphous wild guess
A defined wild guess of something which is not finalised is still miles better than an undefined, implicit, amorphous wild guess
In my experience nobody keeps track of which interfaces were wild guesses. And they look exactly the same to an outsider.
Maybe it's not types, but Python.
Let's consider the ultimate programming language that is able to infer your business requirement before you do or you even knowing what it is. Its only limitation (perhaps by design) is that it just can't write it. It can perform this amazing static analysis of the code and tell things like "hey, the requirement said to return a JSON with the following keys X and Y, but you are missing X. Oh and also you forgot to mail John the patch for that client".
Would programming in this language hard? OF COURSE! This language will constantly complain and wont run your program until you get your requirement exactly right. But I would argue this language is the best language to work with. You might say, "I will never ship a code because it will never be complete!" Well, dear friend, why not just admit that flaw and lower down your business requirement to include those deficiencies?