Python 3: Ten years later(archive.fosdem.org)
archive.fosdem.org
Python 3: Ten years later
https://archive.fosdem.org/2018/schedule/event/python3/
92 comments
In the submission form, have the title field auto populated from the URL.
You don't have to look at many URLs to see how hard that would be.
Not to mention "unless it misleading or linkbait", which is not something code can take care of.
Not to mention "unless it misleading or linkbait", which is not something code can take care of.
Would take about 5 minutes to integrate this... certainly less than it takes to deal with people coming up with their own titles, sometimes without a malicious intent, they just don't know the rules.
https://embed.ly/extract
https://embed.ly/extract
If the "was it worth it?" bit is the focus of the piece, that's actually helpful and informative.
Not all title additions are bad.
Edit: not really the focus, it's more of an outline of an overview of what happened. But still.
Not all title additions are bad.
Edit: not really the focus, it's more of an outline of an overview of what happened. But still.
I think the conversion would have been faster, especially for people who don't care, if the main distributions collaborated and stopped providing Python 2.x; 99% of software writers would just check out how to write their code in Python 3, or move their code to that version. The other 1% would be people who "knew what they're doing" and custom install whatever version they needed.
Imagine if Ubuntu 14.04 (Trusty) didn't bring Python 2. All devs of relevant stuff would have received a shower of upgrade requests. Maybe 2014 was too early to force this, but I'm sure in 2018 it could have been done with Ubuntu 18.04.
I know this looks like work for the sake of upgrading, without any real benefit for most projects. But I guess it's not wrong either; you just couldn't expect that Qt 3.3 (circa 2004) was distributed 10 years later in Ubuntu 14.04 (Qt is just a library, not a language, but it practically transforms the way you write C++ so it's a valid comparison IMO); however here we are in 2018, 10 years after the release of Python 3, and Python 2 is still the default interpreter in Ubuntu 18.04.
Imagine if Ubuntu 14.04 (Trusty) didn't bring Python 2. All devs of relevant stuff would have received a shower of upgrade requests. Maybe 2014 was too early to force this, but I'm sure in 2018 it could have been done with Ubuntu 18.04.
I know this looks like work for the sake of upgrading, without any real benefit for most projects. But I guess it's not wrong either; you just couldn't expect that Qt 3.3 (circa 2004) was distributed 10 years later in Ubuntu 14.04 (Qt is just a library, not a language, but it practically transforms the way you write C++ so it's a valid comparison IMO); however here we are in 2018, 10 years after the release of Python 3, and Python 2 is still the default interpreter in Ubuntu 18.04.
Python 3 is the open source equivalent of Vista - too much potential, but too much time to realize it, and the world doesn’t slow down in the meantime.
I love python. For me, it’s damn near perfect in many ways. But it’s looking more and more like Perl 6 than what it should have been.
The sooner we get to Python 4 (whatever that ends up being), the better, as we need to move past the v2 or v3 question for developers and users.
I love python. For me, it’s damn near perfect in many ways. But it’s looking more and more like Perl 6 than what it should have been.
The sooner we get to Python 4 (whatever that ends up being), the better, as we need to move past the v2 or v3 question for developers and users.
No.
I wish they spent the effort on making Python 2 faster and removing GIL instead.
Why? I teach Python and c++ to A level students. They pick up the basics in a few weeks. C++ is pretty easy. We push our students through the c++ institute exam with few issues.
Choose the right tool for the job. If you need speed, choose c
Choose the right tool for the job. If you need speed, choose c
C++ is pretty easy!? Wow, that's an opinion you often don't hear. :-)
To be fair, we're not coding huge projects in it, just little toy programs.
No one seems to struggle too much though. And the things they struggle with, they would struggle regardless of the language chosen.
No one seems to struggle too much though. And the things they struggle with, they would struggle regardless of the language chosen.
As in "pretty easy to introduce catastrophic errors with".
If you use the right features of C++17 errors are easier to avoid than ever.
The problem is that this elegant subset of C++ is hidden inside ... the rest of C++.
I guess that's the counter example to Python's 2/3 breaking change. C++ just keeps adding new features while retaining all the old ones for backwards compatibility, even if they're awkward in context of the new language.
The problem is that this elegant subset of C++ is hidden inside ... the rest of C++.
I guess that's the counter example to Python's 2/3 breaking change. C++ just keeps adding new features while retaining all the old ones for backwards compatibility, even if they're awkward in context of the new language.
I once read somewhere that C++ would be fine if one just used a sane subset of C++, but the challenge was agreeing on which subset.
agree with choose the right tool, and don't write it yourself. a beginner in c will struggle to beat python + numpy for speed, no matter how "fast" c is
or Fortran
That and Unicode.
Python 3 is worth it just for Unicode handling. As someone using a language that is uses non-ASCII characters, thank you if you're using Python 3 and made your software more internationally compatible.
...or you could just use the unicode type in Python 2; like: all of my Python 2 software handled languages in other character sets correctly (and yes: this was stress tested often as I have had many many users from China and the Middle East). There was absolutely nothing wrong with Python 2 and Unicode, and if anything Python 3 just made a bunch of stuff impossible to do correctly (in particular encodings for filenames) until years later (I think Python 3.4 or something is when they finally got their Unicode shit together; meanwhile, Python 2 worked great).
Except doing the right thing by default is important, and the reality is that Python 3 code is way more likely to handle Unicode correctly, which is what we want.
People constantly complain, but honestly changes like Python 3 have to happen - the alternative is just another completely new language taking over instead.
People constantly complain, but honestly changes like Python 3 have to happen - the alternative is just another completely new language taking over instead.
The only time I had Unicode trouble with python 2 was when a library thought it had to handle encoding. Instead of just piping in bytes on one end and piping out bytes at the other it failed the moment it saw an ü on a system with german text. Unicode certainly has a place, introducing thousands of system specific error and edge cases in what should be a few lines of utility script is not one of them.
I'm less enthused with the defaults than you are. How many people do you think would correctly handle foreign text in this Python 3 exercise:
Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout.
My bet is very few people would write that program correctly.
Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout.
My bet is very few people would write that program correctly.
Ok, I'll bite.
This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial:
https://gist.github.com/avian2/e3968bf933212ee561410b15495e6...
What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.
This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial:
https://gist.github.com/avian2/e3968bf933212ee561410b15495e6...
What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.
I saved this page from the English Wikipedia as input.txt: https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe
C:\Users\local\tmp>python --version
Python 3.7.1
C:\Users\local\tmp>python foo.py Diet Parliament
Traceback (most recent call last):
File "foo.py", line 7, in <module>
text = f.read()
File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\
cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 107: char
acter maps to <undefined>
The way I usually put it is that Python 3 shifted its priorities.
Previously, if you were a UNIX-y scripter writing UNIX-y scripts on your UNIX-y OS, the fact that Python just kind of pretended encoding issues would never exist was a help to you. It adopted the same "everything is ASCII, or at most UTF-8 in the ASCII range, and if it isn't I'll break in cryptic ways" approach as most other UNIX-y scripting things.
If you were doing anything other than UNIX-y scripting on your UNIX-y OS, this easily became a huge nightmare in Python 2. Django went through a massive rewrite early in its history precisely because of this, to ensure that encoding/decoding happened at the boundaries and everything you'd work with inside a Django app was already a Unicode string. And I remember what it was like trying to work on the web before that approach, and what the work to fix it was like.
Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths, in order to free the rest of the Python community from the nightmares inflicted by prioritizing the UNIX-y scripters to the exclusion of everyone else. So yes, you have to do more work. Yes, you have to learn that a file path is actually an opaque bag of bytes that may not be in any actual encoding and thus can never properly decode to a string. Yes, you have to learn to use fsencode() and the surrogateescape handler in order not to blow up your scripts.
But I'm OK with that, because it puts the workload on you when you're using such a system, rather than magically trying to fix it for you at the cost of everyone else's sanity. It also means that you have to learn to write those scripts correctly. Which is more work than what Python 2 required, but not the world-ending apocalyptic horror it's usually presented as (and is, again, mostly the fault of UNIX-y systems doing their old UNIX-y things, not the fault of Python).
Previously, if you were a UNIX-y scripter writing UNIX-y scripts on your UNIX-y OS, the fact that Python just kind of pretended encoding issues would never exist was a help to you. It adopted the same "everything is ASCII, or at most UTF-8 in the ASCII range, and if it isn't I'll break in cryptic ways" approach as most other UNIX-y scripting things.
If you were doing anything other than UNIX-y scripting on your UNIX-y OS, this easily became a huge nightmare in Python 2. Django went through a massive rewrite early in its history precisely because of this, to ensure that encoding/decoding happened at the boundaries and everything you'd work with inside a Django app was already a Unicode string. And I remember what it was like trying to work on the web before that approach, and what the work to fix it was like.
Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths, in order to free the rest of the Python community from the nightmares inflicted by prioritizing the UNIX-y scripters to the exclusion of everyone else. So yes, you have to do more work. Yes, you have to learn that a file path is actually an opaque bag of bytes that may not be in any actual encoding and thus can never properly decode to a string. Yes, you have to learn to use fsencode() and the surrogateescape handler in order not to blow up your scripts.
But I'm OK with that, because it puts the workload on you when you're using such a system, rather than magically trying to fix it for you at the cost of everyone else's sanity. It also means that you have to learn to write those scripts correctly. Which is more work than what Python 2 required, but not the world-ending apocalyptic horror it's usually presented as (and is, again, mostly the fault of UNIX-y systems doing their old UNIX-y things, not the fault of Python).
So, write the program correctly. Show me.
> Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths
Show me how much Python 3 improves this. To expand on the program before, make it a directory named 'input' full of files, and a directory named 'output' to put the processed files in. Print each file name as the corresponding file is processed to indicate progress.
I would be applauding Python if it did make the difficulties with this exercise obvious, but it absolutely does not. The file system APIs return strings, but the strings they return may not be valid Unicode. PEP 383 turned the Python 3 str type into a bag of bytes.
Python tries to sweep encodings under the rug. It makes the encoding a default value all over the place and hides conversions everywhere.
I 100% agree that developers need to think about encodings and handle them in their programs. That's exactly why I hate string handling in Python 3: because rather than making you handle the corner cases, it pretends they don't exist, until they're found one by one by your users.
Python 3 encourages developers to write broken string handling code.
> Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths
Show me how much Python 3 improves this. To expand on the program before, make it a directory named 'input' full of files, and a directory named 'output' to put the processed files in. Print each file name as the corresponding file is processed to indicate progress.
I would be applauding Python if it did make the difficulties with this exercise obvious, but it absolutely does not. The file system APIs return strings, but the strings they return may not be valid Unicode. PEP 383 turned the Python 3 str type into a bag of bytes.
Python tries to sweep encodings under the rug. It makes the encoding a default value all over the place and hides conversions everywhere.
I 100% agree that developers need to think about encodings and handle them in their programs. That's exactly why I hate string handling in Python 3: because rather than making you handle the corner cases, it pretends they don't exist, until they're found one by one by your users.
Python 3 encourages developers to write broken string handling code.
So, write the program correctly. Show me.
You just want to fight someone because you're angry, and I don't do that. No matter what someone writes you'll find a way to argue into it being wrong and then prance around declaring "victory".
(I'd also bet that you probably couldn't do it if I were the one who got to set the evaluation criteria, and you also couldn't pass other "challenges" like writing proper HTTP handling -- the person who gets to grade the challenge always "wins", which is why you want to be the person who grades the challenge)
PEP 383 turned the Python 3 str type into a bag of bytes
PEP 383 provided a way to read certain things -- primarily filesystem paths which can be basically anything -- using an escape mechanism to replace non-decodable bytes with surrogates when decoding to string, which in turn allow losslessly transforming back to the original bag of bytes.
Which is necessary, because there are real filesystems out there that really have paths and names that can never validly decode from any known text encoding. It doesn't turn strings into "bags of bytes"; the resulting str still is an iterable of actual valid Unicode code points.
Python tries to sweep encodings under the rug.
As the saying goes, you can't reason someone out of a position they didn't reason their way into, so I won't try here.
Python 3 encourages developers to write broken string handling code.
Python 3 no longer tries to cover for the random gibberish that's legal to put in filesystem paths, and makes the developer handle it. Are there lots of developers out there who don't realize that filesystem paths can legally contain undecodable garbage? Sure. That's not Python's problem to solve, though; it gives you the surrogateescape handler, and the fsencode helper, and keeps working on things like PEP 538 and PEP 540 to try to give you tools to work around it. But Python can't magically fix the mess that is UNIX locales and bag-of-bytes paths (nothing can, short of burning UNIX down and starting over), and doesn't try to do it for you.
You just want to fight someone because you're angry, and I don't do that. No matter what someone writes you'll find a way to argue into it being wrong and then prance around declaring "victory".
(I'd also bet that you probably couldn't do it if I were the one who got to set the evaluation criteria, and you also couldn't pass other "challenges" like writing proper HTTP handling -- the person who gets to grade the challenge always "wins", which is why you want to be the person who grades the challenge)
PEP 383 turned the Python 3 str type into a bag of bytes
PEP 383 provided a way to read certain things -- primarily filesystem paths which can be basically anything -- using an escape mechanism to replace non-decodable bytes with surrogates when decoding to string, which in turn allow losslessly transforming back to the original bag of bytes.
Which is necessary, because there are real filesystems out there that really have paths and names that can never validly decode from any known text encoding. It doesn't turn strings into "bags of bytes"; the resulting str still is an iterable of actual valid Unicode code points.
Python tries to sweep encodings under the rug.
As the saying goes, you can't reason someone out of a position they didn't reason their way into, so I won't try here.
Python 3 encourages developers to write broken string handling code.
Python 3 no longer tries to cover for the random gibberish that's legal to put in filesystem paths, and makes the developer handle it. Are there lots of developers out there who don't realize that filesystem paths can legally contain undecodable garbage? Sure. That's not Python's problem to solve, though; it gives you the surrogateescape handler, and the fsencode helper, and keeps working on things like PEP 538 and PEP 540 to try to give you tools to work around it. But Python can't magically fix the mess that is UNIX locales and bag-of-bytes paths (nothing can, short of burning UNIX down and starting over), and doesn't try to do it for you.
You're right that I'm frustrated. I'm not even upset at anyone in this thead, but from previous discussions. I appologise for carrying that baggage here. Having my morality questioned when I legitimately want to make programs work correctly for foreign languages has left me... emotional on this subject.
Unix has its own problems, but the program avian wrote works correctly on Linux. Most encoding issues I encountered when working with Python 3 were on Windows.
PEP 383 was making the best of a bad situation without breaking the API again. The real mistake was having the functions return strings in 3.0. The operating system APIs should have returned path objects that require an explicit conversion to string with an explicit error handling mechanism.
Python gives you all the tools you need to do this right, but they're easy to unknowingly use in ways that break on corner cases. A well-defined API should guide you towards the correct solution and should make pitfalls obvious.
In any case, I should probably give it a rest. I work hard to make sure my programs do this stuff right, and I suppose that's all I can really do.
Unix has its own problems, but the program avian wrote works correctly on Linux. Most encoding issues I encountered when working with Python 3 were on Windows.
PEP 383 was making the best of a bad situation without breaking the API again. The real mistake was having the functions return strings in 3.0. The operating system APIs should have returned path objects that require an explicit conversion to string with an explicit error handling mechanism.
Python gives you all the tools you need to do this right, but they're easy to unknowingly use in ways that break on corner cases. A well-defined API should guide you towards the correct solution and should make pitfalls obvious.
In any case, I should probably give it a rest. I work hard to make sure my programs do this stuff right, and I suppose that's all I can really do.
Well as you can see the problem is that open without encoding arguments uses the system default settings (in this case cp1252) which is always a terrible idea regardless of how a language represents strings internally.
Honestly it surprises me how simple and easy it is to understand this trace and I'd agree with the statement that this is a good user experience.
Honestly it surprises me how simple and easy it is to understand this trace and I'd agree with the statement that this is a good user experience.
It's acceptable for developers, but when I say "user friendly" I mean friendly to end users, who may not even know that the program is written in Python. A better message would tell users in plain language what the problem is and how they could fix it (e.g. by setting a flag or an environment variable).
Good for you, but that isn't how it usually worked. Python 3 has made my life easier, but if you really think "There was absolutey nothing wrong with Python 2 and Unicode", I doubt I'll be able to convince you. So I'll simply say again: thank you Python 3 maintainers and developers, it's a huge help to me.
I was always able to work out my unicode trouble in Python 2 ... eventually.
It's better now, less space to screw up.
It's better now, less space to screw up.
hnauz(2)
We where able to delete so much code when we switched to Python 3. When you integrate with especially Windows system converting unknown text was tricky at best.
It’s not that Python 2 couldn’t do it, but the corner cases and error handling was a pain. Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.
Python 3 was a rocky transision, but I never understood the negativity and I’ll be happy to see Python 2 die, so our system can go back to having just one Python interpretter.
It’s not that Python 2 couldn’t do it, but the corner cases and error handling was a pain. Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.
Python 3 was a rocky transision, but I never understood the negativity and I’ll be happy to see Python 2 die, so our system can go back to having just one Python interpretter.
I think this is largely a second system result. If you want correct unicode handling it's not harder in 2.x than in 3.x. The main reason unicode support was more complex in 2.x for the most part is that many APIs supported working with Bytes and on 3.x that burden is pushed to the API consumer.
> Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.
You should not guess on 2.x either. On the other hand in 3.x the interpreter sometimes guesses the encoding for you when working with OS data on unix and that has lead to some absurd bugs when it falls into that surrogate handling fallback code: http://click.palletsprojects.com/en/7.x/python3/#python-3-su...
> Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.
You should not guess on 2.x either. On the other hand in 3.x the interpreter sometimes guesses the encoding for you when working with OS data on unix and that has lead to some absurd bugs when it falls into that surrogate handling fallback code: http://click.palletsprojects.com/en/7.x/python3/#python-3-su...
At first blush you would think so, but I know for a fact that Unicode on Windows is not possible to properly handle below Python 3.4.
Python 3’s Unicode handling has been a huge win for cross-platform deployments in my experience. We run a patched version on 3.3 with some backports from 3.4 on millions of machines, and it is actually possible to support non-Latin filesystem paths, where that was just not possible when we were using 2.6.
In terms of an ecosystem, Python 3 forces casual developers to confront Unicode head on, where 2.x the default is to sweep it under the rug.
I’m extremely grateful that a breaking change was made, even if the execution wasn’t perfect.
Python 3’s Unicode handling has been a huge win for cross-platform deployments in my experience. We run a patched version on 3.3 with some backports from 3.4 on millions of machines, and it is actually possible to support non-Latin filesystem paths, where that was just not possible when we were using 2.6.
In terms of an ecosystem, Python 3 forces casual developers to confront Unicode head on, where 2.x the default is to sweep it under the rug.
I’m extremely grateful that a breaking change was made, even if the execution wasn’t perfect.
Click uses ctypes to retrofit some unicode support on 2.x for what it’s worth.
>on 3.x that burden is pushed to the API consumer
So for most people it's better. The same goes for the guessing encoding part. Sure, you really shouldn't, but sometimes you had to and then you end up with chardet all over the place. Python 3 may do the exact same behind the scenes. The thing is that the Python core developer are much more talented than I am, so now I get to not think to much about it and just build on their work. From the "regular end-user developer" it's better.
So for most people it's better. The same goes for the guessing encoding part. Sure, you really shouldn't, but sometimes you had to and then you end up with chardet all over the place. Python 3 may do the exact same behind the scenes. The thing is that the Python core developer are much more talented than I am, so now I get to not think to much about it and just build on their work. From the "regular end-user developer" it's better.
Random scaremongers. Just look at pypi
Yes. And I guess it could have gone worth. On the other hand, I don't expect the community in a mood to embark on Python 4 any time soon, though it would make sense, in terms of version numbers and time passed.
There are still idiots on Github using Python 2 for new projects. These are usually academics who don't actually share any love for technology. Their code will never be used by anyone.
>Their code will never be used by anyone.
Critical FORTRAN libraries are still the center of certain types of analysis, and they're often used by _everyone_ in the field.
The reason for that stems from two issues:
1. Making accurate simulation libraries for things like weather formation, magnetic field formation or hydraulic fracturing requires crazy in depth knowledge of the systems you are modelling, and takes years of implementing and hypothesis testing to ensure they are accurate enough for academic use.
2. Academics are not software developers, they don't get paid for the software they write, they get paid for papers. They get no reward for rewriting those crazy indepth libraries every decade in the new language de-jour.
This is a perfect example of why the python2 to python3 hard-cutoff transition was a terrible idea. "Just update your software" is something you can say to professional software developers with a straight face. You can't say it to academics, or mechatronics engineers, or real estate agents, or photographers, or _anyone else_ who has a job that benefits from a little bit of software but is mostly about delivering something other than software.
Well, you can, but they're going to ignore you.
Critical FORTRAN libraries are still the center of certain types of analysis, and they're often used by _everyone_ in the field.
The reason for that stems from two issues:
1. Making accurate simulation libraries for things like weather formation, magnetic field formation or hydraulic fracturing requires crazy in depth knowledge of the systems you are modelling, and takes years of implementing and hypothesis testing to ensure they are accurate enough for academic use.
2. Academics are not software developers, they don't get paid for the software they write, they get paid for papers. They get no reward for rewriting those crazy indepth libraries every decade in the new language de-jour.
This is a perfect example of why the python2 to python3 hard-cutoff transition was a terrible idea. "Just update your software" is something you can say to professional software developers with a straight face. You can't say it to academics, or mechatronics engineers, or real estate agents, or photographers, or _anyone else_ who has a job that benefits from a little bit of software but is mostly about delivering something other than software.
Well, you can, but they're going to ignore you.
There has been a pledge by the largest group of Python developers not to mess around with Python 2. So if you want your code to be able to just run forever without any concern for all the new and exciting language ideas being loaded into Python 3 then Python 2 is the logical choice.
I've heard of this - people considering Python 2.7 as an "LTS" version of the language, receiving bug fixes but otherwise remaining unchanged for many years.
I wonder, once 2.7 is EOL in a years time, whether the language developers will consider a "3.x LTS" release?
I wonder, once 2.7 is EOL in a years time, whether the language developers will consider a "3.x LTS" release?
As long as you don’t need to connect to sites using https
Definitely yes, it took some time and had some costs but nowadays python (3) is a much better language. Perhaps it could have been done differently, however it is in the past now.
It's a good example of why JS hasn't had changes that break existing code, so that's good
As an outsider who only uses Python when I need a specific tool that's only available through pip: it's still one of my biggest nightmares.
Every single time I need to install something with pip on MacOS there is this hour-long struggle with figuring out how to get the correct version of Python (2.7 vs 3) working with the correct version of pip working with the correct version of the library itself.
I don't know, I guess people who use Python regularly have figured it out, or maybe it's just a MacOS thing. For me, it's just a major PITA even 10 years after release.
Every single time I need to install something with pip on MacOS there is this hour-long struggle with figuring out how to get the correct version of Python (2.7 vs 3) working with the correct version of pip working with the correct version of the library itself.
I don't know, I guess people who use Python regularly have figured it out, or maybe it's just a MacOS thing. For me, it's just a major PITA even 10 years after release.
I use Ubuntu and it's the exact same thing for me, so I don't think it's MacOS-specific. To be honest, the whole versioning quagmire is the main reason why I don't use Python as one of my main languages, but just as an outsider like you. I like the language but I hate wasting my time with that kind of thing. Say what you will about Java, but you can just slap a jar file into a folder and it will work with the latest version, with zero drama.
On my Debian box I notice I have pip2.7 and python2.7, but also pip3.4, python3.4, pip3.6, python3.6, etc... If I want to install something for python2.7 I just use pip2.7. It looks like on this system python and pip are just symlinks to python2.7 and pip2.7 while python3 and pip3 link to python3.8 and pip3.8
It never seemed that hard to keep track of the different versions.
It never seemed that hard to keep track of the different versions.
I feel like the fundamental issue is that it simply does not make sense for pip to be able to install or upgrade any Python packages managed by the system package manager, any more than it makes sense for it to upgrade the system-level Python interpreter itself. It should just outright not have the ability to do this.
I don't have a Mac and maybe you've already tried this, but just in case, you can always do python2 -m pip install ... or python3 -m pip install ... to make sure it's running within that particular Python installation. You can use the --user flag to install to your user directory as well, if the system level is annoying.
why is it that the pip command can't handle what it's supposed to do, instead you have to resort to `python3 -m pip install`?
Hell, it even breaks itself:
pip install --upgrade pip # => pip unusable
https://github.com/pypa/pip/issues/5447
https://github.com/pypa/pip/issues/5221
etc
Hell, it even breaks itself:
pip install --upgrade pip # => pip unusable
https://github.com/pypa/pip/issues/5447
https://github.com/pypa/pip/issues/5221
etc
Well pip is only one command, so it can't simultaneously work for both Python 2 and Python 3. And if you happen to have another installation of Python or pip in your path (e.g. at the user-level) but not have the other one there, then they'd affect different installations entirely. When you run it via Python you know it's affecting exactly the same installation.
Upgrading pip itself is kind of a special case I think. You don't want to upgrade stuff managed by a system package manager using pip; their work will conflict with each other. This includes pip itself. I think this is kind of a mess and I don't know of a great way to deal with it other than just remembering not to upgrading system packages using pip (yes this includes pip itself, and means you should avoid trying to upgrade via --user if it's installed on the system). On Windows where I am it's ironically less annoying since there's no system-level package manager to worry about... go figure.
Upgrading pip itself is kind of a special case I think. You don't want to upgrade stuff managed by a system package manager using pip; their work will conflict with each other. This includes pip itself. I think this is kind of a mess and I don't know of a great way to deal with it other than just remembering not to upgrading system packages using pip (yes this includes pip itself, and means you should avoid trying to upgrade via --user if it's installed on the system). On Windows where I am it's ironically less annoying since there's no system-level package manager to worry about... go figure.
How would pip know if you want python3 or 2?
On Mac, the system uses python2 so pip and python link to py2.
You can change the system -not fun- or wait for apple to upgrade to py3. Ive been waiting a long time.
There are tools like conda that do manage py2 and py3 projects living in harmony.
On Mac, the system uses python2 so pip and python link to py2.
You can change the system -not fun- or wait for apple to upgrade to py3. Ive been waiting a long time.
There are tools like conda that do manage py2 and py3 projects living in harmony.
Nope, pip2, pip3. I upgrade all the time no problems, lin and win however.
with homebrewed python, i use 'pip2' or 'pip3' and haven't had an issue (or better yet, a virtualenv with the appropriate interpreter). don't use apple's default python. hope that helps!
i highly recommend pyenv to resolve this issue on Mac OS and Linux:
https://github.com/pyenv/pyenv
https://github.com/pyenv/pyenv
Seconded. Pyenv has made life much better for me when using system-wide py utils.
Yep! Pyenv for Python, rbenv for Ruby, NVM for node.
My experience using nvm has been quite annoying.
What you could do with simple npm command now doesn't work unless you give additional command lines when it comes to cron and other non interactive environments like process monitoring to restart the process.
What you could do with simple npm command now doesn't work unless you give additional command lines when it comes to cron and other non interactive environments like process monitoring to restart the process.
Create a virtualenv and install the version you need of Pip there
Or use Anaconda, which is your own Python "Disneyland" but it is hard to interface with external infrastructure
Or use Anaconda, which is your own Python "Disneyland" but it is hard to interface with external infrastructure
Out of curiosity do you use virtual environments? I had this same problem until I became very liberal with my use of virtual environments.
For anything larger than a medium-sized single-file script, I use pipenv. It's basically a tie-in between pip and virtualenv, and in my opinion is ideal for creating isolated environments quickly, intuitively, and reproducibly. I haven't had Python frustrations since starting to use pipenv.
I was not paid for this message :)
I was not paid for this message :)
Never had any issue with python on mac os. I install python with homebrew and then use pip3 and python3.
I always use "python -m pip install TOOL_NAME" or "python3 -m pip install TOOL_NAME".
$ pip3 install foo
Or pip3.6 etc
--user is useful also
Or pip3.6 etc
--user is useful also
No, IMO. Don't get me wrong, Python 3 is a better language than Python 2. I just don't think it's that much better that it was worth breaking backwards compatibility and stalling the language for several years.
Python 3 did enough to break everyone's existing code, but nowhere near enough to make that worthwhile.
It's common for other language communities to consider the Python 2/3 transition as an example of "what not to do".
Python 3 did enough to break everyone's existing code, but nowhere near enough to make that worthwhile.
It's common for other language communities to consider the Python 2/3 transition as an example of "what not to do".
How hard is it to simply keep one's code up to date? I'm pretty sure there weren't any radically different paradigms introduced in Python 3.
Somehow the world of JavaScript has been able to quickly adapt to a rapidly changing language and the community is hardly fractured along the lines of language. Adopting new language features and browser APIs is just a matter of fact. What's different about Python?
Somehow the world of JavaScript has been able to quickly adapt to a rapidly changing language and the community is hardly fractured along the lines of language. Adopting new language features and browser APIs is just a matter of fact. What's different about Python?
JavaScript has been stuck with an ancient version because of browser adoption like IE6.
There's a good reason people wanted to migrate to newer JS and I never heard people say ES6 got something wrong over ES5.
There's a good reason people wanted to migrate to newer JS and I never heard people say ES6 got something wrong over ES5.
I skipped 3.0-3.3 and used the logging module in every nontrivial program, so py3 never broke anything of consequence to me.
I think Python 3 primarily showed that breaking backwards incompatibility across an entire ecosystem is worse than one could have predicted. Even today Python 2 is far from dead.
On the other hand the core development team "outsourced" many of the issues that people actually want to have solved to the community like packaging and distribution.
Python's biggest downfall is a highly academic approach with slow release cycles which makes large feature development hard and detached from the real world. Python 3's unicode model is completely outdated but was supposed to be the big feature of the version. While that was happening however the rest of the world started moving over to UTF-8. There was no feedback loop that could have enabled Python to be UTF-8 based. It was released and then decisions were locked in.
Similar things happened in asyncio. New versions of it only land with every major version of the language which meant that a lot of asyncio only became really useful with 3.6 and 3.7 which was years after it was initially created.
I have very high hopes that the new leadership of the language will be able to tackle some of these problems. Python 3 is way less backwards incompatible to 2.7 now than it was 10 years ago. The two versions moved closer to each other and today it's pretty easy to support both from a single codebase. With faster iteration we might have been able to cut this time short by 5 years or more
On the other hand the core development team "outsourced" many of the issues that people actually want to have solved to the community like packaging and distribution.
Python's biggest downfall is a highly academic approach with slow release cycles which makes large feature development hard and detached from the real world. Python 3's unicode model is completely outdated but was supposed to be the big feature of the version. While that was happening however the rest of the world started moving over to UTF-8. There was no feedback loop that could have enabled Python to be UTF-8 based. It was released and then decisions were locked in.
Similar things happened in asyncio. New versions of it only land with every major version of the language which meant that a lot of asyncio only became really useful with 3.6 and 3.7 which was years after it was initially created.
I have very high hopes that the new leadership of the language will be able to tackle some of these problems. Python 3 is way less backwards incompatible to 2.7 now than it was 10 years ago. The two versions moved closer to each other and today it's pretty easy to support both from a single codebase. With faster iteration we might have been able to cut this time short by 5 years or more
> Python 3's unicode model is completely outdated but was supposed to be the big feature of the version. While that was happening however the rest of the world started moving over to UTF-8.
I’m not sure what you mean. Are you suggesting that Python use UTF-8 encoded strings internally instead of decoded Unicode codepoints like it does currently?
I’ve worked on Perl that uses that approach and it’s an absolute nightmare. All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely. You get hard-to-find bugs when some code somewhere introduces invalid UTF to a string. The only real benefit you get is slightly smaller memory footprint when dealing mostly with English text.
I much prefer Python’s decode-process-encode approach.
I’m not sure what you mean. Are you suggesting that Python use UTF-8 encoded strings internally instead of decoded Unicode codepoints like it does currently?
I’ve worked on Perl that uses that approach and it’s an absolute nightmare. All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely. You get hard-to-find bugs when some code somewhere introduces invalid UTF to a string. The only real benefit you get is slightly smaller memory footprint when dealing mostly with English text.
I much prefer Python’s decode-process-encode approach.
The python nightmare is purely performance due to its use of the UCS-2 or 4 encoding (wchar_t without surrogate pairs). UTF-8 was ruled out for its inability for direct indexing, but this proved wrong. 4x longer string buffers trash the cache much more than the needed CPU cycles for utf-8 decoding.
perl's problem is that they didn't make room for the original encoding, it's only latin-1 or utf-8 from everything else. so the context needs to store it, like IO, process. and for performance they silently loose the utf-8 flag for 128—255 codepoints.
but all this is still better than the libc/POSIX design failure to use ENV vars, where the encoding is global, not localized. every single string API needs to do a getenv, and its interpretation is action at a distance.
perl's problem is that they didn't make room for the original encoding, it's only latin-1 or utf-8 from everything else. so the context needs to store it, like IO, process. and for performance they silently loose the utf-8 flag for 128—255 codepoints.
but all this is still better than the libc/POSIX design failure to use ENV vars, where the encoding is global, not localized. every single string API needs to do a getenv, and its interpretation is action at a distance.
> it's only latin-1 or utf-8
There’s no other way, is it? You basically have to have the complete string library re-implemented for each supported multi-byte encoding.
There’s no other way, is it? You basically have to have the complete string library re-implemented for each supported multi-byte encoding.
No. Foreign encodings are internally converted to utf-8 via the Encoding library and IO layers.
The string library is only done for latin-1 and utf-8.
The only trouble is that you have to remember the original encoding, if you need that information. It's stored nowhere. E.g. to write to a file in the original encoding you have to open it with the correct IO layer. perl6 got that fixed.
The only trouble is that you have to remember the original encoding, if you need that information. It's stored nowhere. E.g. to write to a file in the original encoding you have to open it with the correct IO layer. perl6 got that fixed.
> I’m not sure what you mean. Are you suggesting that Python use UTF-8 encoded strings internally instead of decoded Unicode codepoints like it does currently?
Correct. Because the current approach completely fell apart when people started using emojis. The internal encoding in too many cases is now always UCS4 which is very wasteful.
> All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely.
Go, Rust and Swift all show that using UTF-8 internally is not hard. Not sure why you think its hard. I have written a lot of code in those languages now and I feel much happier with their unicode support than Python's.
Correct. Because the current approach completely fell apart when people started using emojis. The internal encoding in too many cases is now always UCS4 which is very wasteful.
> All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely.
Go, Rust and Swift all show that using UTF-8 internally is not hard. Not sure why you think its hard. I have written a lot of code in those languages now and I feel much happier with their unicode support than Python's.
> Even today Python 2 is far from dead
It's not dead, but as of now, it's mostly relegated to legacy code and a dwindling number of holdout programmers who refuse to adapt based on some misguided principle. It was a long and painful process, but existing libraries are dropping support for Python 2, and new libraries are created daily that will never support Python 2.
That said, I still regularly interact with people who are new to Python but have been indoctrinated somewhere with the belief that they must use Python 2.7. They've heard it's more stable, faster, or Google uses it. We discuss some of the actual tradeoffs, and they're quickly disabused of the notion that Python 2.7 is a good option for new Python projects today.
But I totally agree, some Python 3 additions (I'm looking at you, asyncio) have been handled in an unfortunate way.
It's not dead, but as of now, it's mostly relegated to legacy code and a dwindling number of holdout programmers who refuse to adapt based on some misguided principle. It was a long and painful process, but existing libraries are dropping support for Python 2, and new libraries are created daily that will never support Python 2.
That said, I still regularly interact with people who are new to Python but have been indoctrinated somewhere with the belief that they must use Python 2.7. They've heard it's more stable, faster, or Google uses it. We discuss some of the actual tradeoffs, and they're quickly disabused of the notion that Python 2.7 is a good option for new Python projects today.
But I totally agree, some Python 3 additions (I'm looking at you, asyncio) have been handled in an unfortunate way.
I don’t tend to find much emoji in application code getting manipulated. Perhaps it makes some difference in user messages, however text space is dwarfed by images or even audio.
Makes me wonder how much a problem this is and where.
Makes me wonder how much a problem this is and where.
It's interesting to watch Java doing the same thing at the moment (IHMO). They got obsessed with the runtime size 10 years ago and in solving that we now have a module system that breaks backwards compatibility. But many of the problems we do have are either not being addressed or are not getting serious treatment.
Already we have much of the community still sitting on Java 8 even as the new long term release JDK11 has been made. It wouldn't surprise me at all if JDK8 becomes the Python2 of the java world: 10 years from now, there will still be whole segments of the community using it.
Already we have much of the community still sitting on Java 8 even as the new long term release JDK11 has been made. It wouldn't surprise me at all if JDK8 becomes the Python2 of the java world: 10 years from now, there will still be whole segments of the community using it.
Just one word: typing [1].
Makes life a lot easier.
[1]: https://docs.python.org/3/library/typing.html
Makes life a lot easier.
[1]: https://docs.python.org/3/library/typing.html
Two words: typing, mypy [1]
The editor support that I get from typing/mypy is fantastic! It feels like writing in a statically typed language. To me this is by far the most important feature of Python3. It's a godsend for larger/complicated code bases.
[1]: https://github.com/python/mypy
The editor support that I get from typing/mypy is fantastic! It feels like writing in a statically typed language. To me this is by far the most important feature of Python3. It's a godsend for larger/complicated code bases.
[1]: https://github.com/python/mypy
Yes, you are right. I'm also using mypy along with pylint, yapf and isort. Helps keeping my python codes clean
It worth it, not for the language itself but for teaching everyone a good lesson - don't destroy the backwards compatibility when experimenting your incremental ideas.
Nope.
But it survived, which is the important thing. I thought 3 would kill Python, but it has survived and that is a very good thing.
Anecdotally, non-nerds fins python easier to read than other popular languages. It’s become the go-to language for a bunch of domains that reach outside core nerding (data analysis, education, DSL like stuff) and I predict Python will therefore be widely used in 2050, and probably be popular aged 100.
But it survived, which is the important thing. I thought 3 would kill Python, but it has survived and that is a very good thing.
Anecdotally, non-nerds fins python easier to read than other popular languages. It’s become the go-to language for a bunch of domains that reach outside core nerding (data analysis, education, DSL like stuff) and I predict Python will therefore be widely used in 2050, and probably be popular aged 100.
Python almost became my main language, but I moved to another language when I saw it was stuck at 2.7. Another reason was the lambda's only allowing 1 line instead of the flexible closures in so many other languages. And my expectations for future improvements were very low as the core maintainer looked quite stubborn to me.
Pretty much how I ended up doing most of my coding in Groovy. I essentially wanted Python. But Python wasn't a good enough Python. (being honest, other things attracted me to Groovy as well - Java ecosystem etc, but it's interesting to me that in the larger picture, I'm sort of using Groovy as a proxy for a Python without Python's flaws).
Titles are by far the strongest influence on discussion, so this rule is much more important than it seems.
https://news.ycombinator.com/newsguidelines.html
(Submitted title was "Python 3 Is Now 10 Years Old Was It Worth It?".)