Google is working on Python 3 for App Engine Standard Environment(issuetracker.google.com)
issuetracker.google.com
Google is working on Python 3 for App Engine Standard Environment
https://issuetracker.google.com/issues/35876441#comment104
57 comments
Feature development of GAE (especially the Standard Environment) never stopped. Some of the features require multi quarters of work (e.g. https://cloudplatform.googleblog.com/2018/05/Increase-perfor...), some are more enterprise-focused, some are not user visible but we improve the reliability with a new stack behind the curtain. We are changing the engine while flying (pun not intended).
For various reasons, supporting new runtimes was never an easy task, even for small version bumps. This has changed drastically, starting with Java8 runtime, to the recently announced NodeJS beta, and to the closed alpha of Python 3 (remember this is Python 3.7). If you followed recent announcements from Google you probably can connect the dots.
I think we are pretty open on what we are working on to our customers, as long as an NDA is signed. If you are interested be sure to let our cloud support team know!
Disclaimer: I'm on App Engine SRE team.
For various reasons, supporting new runtimes was never an easy task, even for small version bumps. This has changed drastically, starting with Java8 runtime, to the recently announced NodeJS beta, and to the closed alpha of Python 3 (remember this is Python 3.7). If you followed recent announcements from Google you probably can connect the dots.
I think we are pretty open on what we are working on to our customers, as long as an NDA is signed. If you are interested be sure to let our cloud support team know!
Disclaimer: I'm on App Engine SRE team.
I mean things have been pretty bad for GAE standard in Python for a very long time. It looks so dead I was actually expecting Google to completely remove it in 2 years or so.
I work on a project on GAE standard and the platform was chosen before I started. I would say this was the worst possible solution to chose and the biggest cost for us right now.
The most annoying point is that it is actually not easy at all to switch to the Flexible environment, because libraries are completely different.
Examples of troubles : - new google-cloud libraries run in the flexible environment but not in the standard one (ex: google-cloud-pubsub doesn't run in the standard env after 0.27, and the doc for 0.27 is dead: https://google-cloud-python.readthedocs.io/en/0.27.0/pubsub/...) - traditional standard env libraries do not run in the flexible env. The most typical example is NDB for the Google datastore.
Which means that migrating from the standard to the flexible env requires you to rebuild EVERYTHING that touches Google services... Great, then I'd better just switch to AWS instead!
What's moer, flexible env is nice but for some reason it is not possible to have basic scaling anymore, nor is it possible to scale down to 0 when needed, which makes modular apps much more costly (previously we would have several services and when unused they just scaled down to 0).
On top of that we just watched services get deprecated one after the other (mail service, then the datastore backup etc.) and old troubles not getting fixed (like memory problems with queries).
I work on a project on GAE standard and the platform was chosen before I started. I would say this was the worst possible solution to chose and the biggest cost for us right now.
The most annoying point is that it is actually not easy at all to switch to the Flexible environment, because libraries are completely different.
Examples of troubles : - new google-cloud libraries run in the flexible environment but not in the standard one (ex: google-cloud-pubsub doesn't run in the standard env after 0.27, and the doc for 0.27 is dead: https://google-cloud-python.readthedocs.io/en/0.27.0/pubsub/...) - traditional standard env libraries do not run in the flexible env. The most typical example is NDB for the Google datastore.
Which means that migrating from the standard to the flexible env requires you to rebuild EVERYTHING that touches Google services... Great, then I'd better just switch to AWS instead!
What's moer, flexible env is nice but for some reason it is not possible to have basic scaling anymore, nor is it possible to scale down to 0 when needed, which makes modular apps much more costly (previously we would have several services and when unused they just scaled down to 0).
On top of that we just watched services get deprecated one after the other (mail service, then the datastore backup etc.) and old troubles not getting fixed (like memory problems with queries).
The technique behind standard and flex is large enough that we simply cannot provide consistent experience between these two. In fact these two products are supported by different teams. While the inconsistency is unfortunate, we are fully aware of this issue and are working towards a solution. Please expect some announcements from us.
Service deprecation is not taken lightly, and we guarantee 12 months per GCP deprecation policy (https://cloud.google.com/terms/deprecation). Of the services you mentioned, I don't think mail API is officially deprecated (per https://cloud.google.com/appengine/docs/deprecations/), and Datastore Backups is superseded by Managed Export (https://cloud.google.com/datastore/docs/export-import-entiti...).
Not sure about the memory problems you mentioned though.
Disclaimer: I'm on App Engine SRE team.
Service deprecation is not taken lightly, and we guarantee 12 months per GCP deprecation policy (https://cloud.google.com/terms/deprecation). Of the services you mentioned, I don't think mail API is officially deprecated (per https://cloud.google.com/appengine/docs/deprecations/), and Datastore Backups is superseded by Managed Export (https://cloud.google.com/datastore/docs/export-import-entiti...).
Not sure about the memory problems you mentioned though.
Disclaimer: I'm on App Engine SRE team.
This is a fascinating post as it gives away something about the Google GAE culture perhaps. Some examples:
"multi quarters" of work is odd phraseology. Does the GAE team prefer smaller, easier to schedule, features to longer more difficult projects to keep pace with development of the underyling toolset?
"pretty open" "as long as you sign an NDA" is borderline oxymoronic.
"enterprise focused" speaks for itself.
"multi quarters" of work is odd phraseology. Does the GAE team prefer smaller, easier to schedule, features to longer more difficult projects to keep pace with development of the underyling toolset?
"pretty open" "as long as you sign an NDA" is borderline oxymoronic.
"enterprise focused" speaks for itself.
> "pretty open" "as long as you sign an NDA" is borderline oxymoronic.
> "enterprise focused" speaks for itself.
GAE is a business and has a niche. Perhaps they'd really be dead if they didn't focus on a set of paying customers. NDAs are the norm for enterprise and are reasonable given that they want to maintain their niche.
It's less reasonable for me to comment on their development/deployment methodology since I don't work at Google, but multiple quarters of work doesn't seem unusual for getting products like these deployed across multiple global data centers and achieve the kind of reliability required by these customers.
GAE is a business and has a niche. Perhaps they'd really be dead if they didn't focus on a set of paying customers. NDAs are the norm for enterprise and are reasonable given that they want to maintain their niche.
It's less reasonable for me to comment on their development/deployment methodology since I don't work at Google, but multiple quarters of work doesn't seem unusual for getting products like these deployed across multiple global data centers and achieve the kind of reliability required by these customers.
The Google career page (1) indicates that they are hiring a lot of people in that area, so we might see some fast paced improvements there.
1: https://careers.google.com/jobs#t=sq&q=j&li=20&l=false&j=clo...
1: https://careers.google.com/jobs#t=sq&q=j&li=20&l=false&j=clo...
What have AWS dropped RE Amazon Linux 2? (I thought python 3 was supported there via Amazon Linux Extras.)
Reminder that Python 2 dies in 18 months.
https://pythonclock.org
https://pythonclock.org
It will likely remain around for longer, since depending on unsupported software doesn't immediately kill a project. When old Python 2 code hits a bug in the CPython interpreter and nobody is around to fix it, there'll finally be an incentive to migrate away (maybe to Python 3, maybe to another language), but not before that.
> It will likely remain around for longer, since depending on unsupported software doesn't immediately kill a project.
It also depends on how long companies like Redhat support it (CentOS 6 comes with Python 2.6 and CentOS 7 comes with Python 2.7).
It also depends on how long companies like Redhat support it (CentOS 6 comes with Python 2.6 and CentOS 7 comes with Python 2.7).
True, however lots of large corporates won't use an unsupported version if the vendor drops it.
There, the vendor will be redhat or ... who will be supporting it longer. What enterprises mean by support isn't provided by the PSF anyway.
They support their products 10 years, or indefinitely if you pay extra for it, so Redhat will support Python 2 until June 30, 2024.
If I understand it correctly, that means CentOS will too.
They support their products 10 years, or indefinitely if you pay extra for it, so Redhat will support Python 2 until June 30, 2024.
If I understand it correctly, that means CentOS will too.
The PSF might be trying to murder Python 2, but it can't do it any more than GCC could kill c99 for c11.
I look forward to using c99 and Python 2.7 for many years to come. There are more then enough Companies and Devs to keep Python 2 ticking over forever.
I look forward to using c99 and Python 2.7 for many years to come. There are more then enough Companies and Devs to keep Python 2 ticking over forever.
But.... Why?
My reaction as well.
I understand continuing to build something already written in 2.7. However, building new with 2.7 just seems contrarian.
My problem is I'm rarely building something new -- I have various libraries and utilities in python 2, and will extend and use them in my research, along with coauthors in other universities. Therefore, having a "big bang" change to Python 3 is very difficult.
In practice I imagine some day some PhD student will take it upon themselves to rewrite a significant chunk of libraries to be python 3 compatible, and then we could move on from there, but I wouldnt want to force anyone to spend the time doing a massive Python 3 rewrite.
In practice I imagine some day some PhD student will take it upon themselves to rewrite a significant chunk of libraries to be python 3 compatible, and then we could move on from there, but I wouldnt want to force anyone to spend the time doing a massive Python 3 rewrite.
I'm curious whether you've looked at __future__ and libraries like six? I've found it quite easy to write Python 2/3 compatible code with them and a little planning. I like this approach to maintenance of legacy libraries as it at least prevents making the problem worse.
GCC and clang are probably the only C compilers in the world that care about C11, so it is quite hard to kill C99 for C11.
Last I checked MSVC couldn't even do C99 properly yet.
Because it is called Visual C++ and Microsoft has openly stated that they are compliant to the extent required by ISO C++, which is the future of systems programming on Windows, alongside .NET Native for some use cases.
ISO C++14 requires C99 library compatibility, ISO C++17 requires C11 library compatibility only when C++ doesn't already offer better alternatives.
For C developers there is clang/gcc integration in Visual Studio.
"Modern C++ Game Development on Microsoft Platforms"
https://www.youtube.com/watch?v=on8HMZdAXaw
ISO C++14 requires C99 library compatibility, ISO C++17 requires C11 library compatibility only when C++ doesn't already offer better alternatives.
For C developers there is clang/gcc integration in Visual Studio.
"Modern C++ Game Development on Microsoft Platforms"
https://www.youtube.com/watch?v=on8HMZdAXaw
True, but following the Python model, GCC and Clang could just threaten people with the death of C99 for 5 years or so, then in some release just disable them.
Of course, that would be much less painful than Python 3, because C11 is (a) almost entirely compatible with C99, and (b) not noticeably slower.
Of course, that would be much less painful than Python 3, because C11 is (a) almost entirely compatible with C99, and (b) not noticeably slower.
AFAIK python 3.5 and 3.6 are significantly faster than 2.7 and uses less memory due to dictionary optimizations. I don’t think speed has been a real argument in a while. Also, python 3 is almost entirely compatible with python 2 (although C is probably even more so).
Everyone would have migrated to python 3 years ago if it was almost entirely compatible with python 2. It's just not.
C and Python's compatibility aren't comparable.
It's quite hard to write a C89 program that isn't C99. Basically any Python 2 program that uses 'print' won't be valid Python 3. Even after fixing that, I always find a bunch of string related problems.
It's quite hard to write a C89 program that isn't C99. Basically any Python 2 program that uses 'print' won't be valid Python 3. Even after fixing that, I always find a bunch of string related problems.
I was given to understand that Python 3 got past after 3.5 or so?
Not only did it get faster as a drop in replacement (assuming you thought ahead at all sometime in the last 10 years), but if you can make use of the new async keywords you can get much faster code after a slight refactor.
C99 is a language standard. Python 2 is a language and the CPython runtime, and a massive standard library. They're barely comparable.
What it certainly true is that it will take exponentially more work to keep Python 2 alive. They're not allowed to use the name so that's an instant usage drop-off. And many major third party libraries are dropping support too.
You're probably right though. You'll probably get to use Pytwo (or whatever they call its) for a few more years yet.
What it certainly true is that it will take exponentially more work to keep Python 2 alive. They're not allowed to use the name so that's an instant usage drop-off. And many major third party libraries are dropping support too.
You're probably right though. You'll probably get to use Pytwo (or whatever they call its) for a few more years yet.
I wouldn't be surprised that vendors will still have to do the fixes to Python 2 for many years past that date, and that a kind of "fork" will keep the 2.7 branch decently alive much longer. It's just the current "official" maintainers that announce the cutoff date for their work.
Another problem is 'semi-abandoned' code. For example the Open Source program QGIS used to use Python2 is its main plug in language but recently switched to Python3. The end result is the most plug-ins don't work any more. And since many of the plug-ins where useful little niche scripts that someone wrote years ago and then stopped updating/supporting once they where done, the chance that most of them will see a port is tiny.
Doesn't mean that python 2 programs will stop working. I know developer of Calibre has stated that he will maintain the python 2 codebase himself. And is actually happy that python 2 development will be out of the current maintainers hands.
Appengine standard is amazing. I hope they continue to support 2.7 as well.
Just in case you're like me and had to Bing what the difference is between Docker and Google App Engine, it's summed up really well in this Stack Overflow answer:
https://stackoverflow.com/a/49950687/3799
I wonder, though, if future development should target Docker/K8S and GAE should only be the domain of legacy applications.
https://stackoverflow.com/a/49950687/3799
I wonder, though, if future development should target Docker/K8S and GAE should only be the domain of legacy applications.
> the difference is between Docker and Google App Engine
Neither the question nor the answer in your link compare Docker and App Engine.
Docker is a containerization platform, whereas App Engine is a PaaS (platform as a service). Their functions are wholly different.
Very, very loosely; Docker is about building and running an individual package of some software. Basically, bundling software and running bundles. PaaS are platforms for developing, running, and managing (often pre-bundled) software.
Neither the question nor the answer in your link compare Docker and App Engine.
Docker is a containerization platform, whereas App Engine is a PaaS (platform as a service). Their functions are wholly different.
Very, very loosely; Docker is about building and running an individual package of some software. Basically, bundling software and running bundles. PaaS are platforms for developing, running, and managing (often pre-bundled) software.
Would it be more accurate to compare Google App Engine (PaaS) to Azure Web Apps or App Service (also PaaS)? Docker is in a weird spot of not really being PaaS but yet too light to be considered IaaS... I tend to think of it as sufficiently PaaS-like except it runs on everyone's cloud as well as my laptop.
Google also committed to releasing an update to its App Engine Gw Basic and Turbo Pascal environments, “some time in the 2021 timeframe”.
Yet another example to throw in the face of people who say that the Python 2/3 problem is solved.
at 10 years birthday of app engine
and 10 years birthday of python3
I like to remind people that Python 3 was already more than a year old when Apple announced the first iPad. I'm not sure anyone should be getting a cookie for 'working on' Py3 support more than nine years later. Though I do understand that GAE has supported Py 3 for a while in its more configurable service.
True, though I have to say it only got usable around 3.4 (which still is 4y.o. at this point)
What did 3.4 give you that 2.7 already had? I have found bundled pip and asyncio useful, but I was using Py3 from 3.2, successfully. It didn't suddenly become usable at 3.4 for me.
I think what happened was that 3.2 was the first "real" release, when a lot of .0 bugs were shaken out; the 'u' prefix was added back around that time too, iirc.
This resulted in more developers actually porting their libraries, an effort that had been basically non-existent before. The result was that the 3.4 ecosystem actually got mature enough that Real Work could get done, and more end-users started moving.
IIRC 3.3 also had some big performance problems that were sorted in 3.4, but I cannot remember the details.
Personally, I did very little with 3.2 but actually built Real Stuff only with 3.4.
This resulted in more developers actually porting their libraries, an effort that had been basically non-existent before. The result was that the 3.4 ecosystem actually got mature enough that Real Work could get done, and more end-users started moving.
IIRC 3.3 also had some big performance problems that were sorted in 3.4, but I cannot remember the details.
Personally, I did very little with 3.2 but actually built Real Stuff only with 3.4.
Yeah, pip. You could install it manually, but to be honest, pip and virtualenv are fundamental today (I mean, before containers, but still)
To be fair: PEP 3333 that addresses WSGI support with Python 3 Unicode was initially drafted in September 2010. I would consider that to be the first day we can write a web server in Python 3.
For anyone confused, currently you have a choice of either "Standard Environment", which is Python 2.7 only, or "Flexible Environment" which allows you to choose Python 2.7, Python 3.6, or your own Python runtime [1][2]. This is about supporting Python 3 in the "Standard Environment". It's unclear whether the intention is to replace Python 2 or allow a choice?
[1] https://cloud.google.com/appengine/docs/python/ [2] https://cloud.google.com/appengine/docs/flexible/python/runt...
[1] https://cloud.google.com/appengine/docs/python/ [2] https://cloud.google.com/appengine/docs/flexible/python/runt...
tl;dr Standard environment apps "scale nearly instantaneously to handle huge traffic spikes", "cost-effective for applications that have significant periods where they are not serving traffic", use reasonably limited standard API + proprietary API
More importantly standard scales down to 0 while flexible needs a minimum of 1 instance running at all times. So when just prototyping/developing or creating a small app only you or a few users will use it's very cost effective to use standard.
I love the new movement in on GAE Standard. With GAE Flexible, GKE, Cloud SQL, etc there's actually a growth path for something that out grows it. When Standard first launched, that additional complexity meant a rewrite.
Can the Java runtime implement the full Servlet 3.1 spec now? It's 5 years old (and most Scala http libs don't support blocking Servlets anymore).
Can the Java runtime implement the full Servlet 3.1 spec now? It's 5 years old (and most Scala http libs don't support blocking Servlets anymore).
I tried Appengine when it first came out but Google never seemed to support it much so I gave up on it.
What lead to this whole Python 2 and 3 divide?
I mean, other languages have new versions too...
I mean, other languages have new versions too...
Python 1 didn't know about Unicode, so characters were bytes and things were simple (for latin1 users).
Python 2 bolted on Unicode support, so now you had two kinds of strings (old-style strings and Unicode strings) with implicit conversions between them (that would blow up spectacularly when you least expected it) and overall a huge mess.
Python 3 removes the old-style strings (replacing them with the "bytes" type and explicit conversions), so that we once again have only one type of string, and once again things are simple. This clean-up unfortunately means that code relying the Python 2 string type confusion will break.
Python 2 bolted on Unicode support, so now you had two kinds of strings (old-style strings and Unicode strings) with implicit conversions between them (that would blow up spectacularly when you least expected it) and overall a huge mess.
Python 3 removes the old-style strings (replacing them with the "bytes" type and explicit conversions), so that we once again have only one type of string, and once again things are simple. This clean-up unfortunately means that code relying the Python 2 string type confusion will break.
Well, at least someone tried to clean things up.
Other languages simply carry these bugs around forever.
Other languages simply carry these bugs around forever.
The transition from Python 2 into Python 3 was badly botched, and for many years. First of all, they confused language with implementation, so the Python 3 interpreter cannot handle the Python 2 language while the Python2 interpreter cannot handle the Python 3 language (I refer to the main C interpreter, not others like PyPy). The automated tool "2to3" wasn't reliable on real-world programs, either. For many years it was impractical to write programs or libraries that worked on both 2 and 3. To write any Python 3 code, ALL of your transitively-dependent libraries had to convert to Python 3, and effectively simultaneously. By itself, that was rediculous. In addition, Python 3 doesn't add any radically new capabilities over Python 2, so it was basically all pain and no gain. Python 3's new approach to strings is conceptually clean, but it struggles with the messiness of the real world and the Python 3 developers had to add a lot of hacks to make it sort-of work. For example, file names are not strings of characters on Unix/Linux, there is no standard encoding for stdin, and real-world data doesn't always follow any particular encoding and there's often no need to require it. Python 3 was effectively ignored by most programmers for at least 5 years after its release, because it created more problems than it solved.
What changed is that Python 3 was modified, and libraries were added to both Python 2 and 3, so that it started to became practical to write code that could work on both 2 and 3. The result: instead of having to transition your entire program and all its dependencies simultaneously, people could work incrementally, making little improvements here and there so that programs would keep working in 2, and eventually work in 3 as well. Various hacks were also created so that Python 3 could handle stuff like filenames (!).
It would have been much better had the Python 3 developers made it easier to transition from 2 to 3 from the start, but it's finally easier. It's still somewhat painful; mercurial still hasn't transitioned, and it's not clear that they ever will. I'm unhappy about what the Python developers did earlier, but I think they've made good strides in trying to finally make it much easier to transition from Python 2 to Python 3.
The main takeaway here should be, "make it as PAINLESS as POSSIBLE for your existing users to transition to a new version". It's okay to change APIs, but use different names or use versioned APIs. It's okay to change other things, but make it easy for people to modify parts of their program at a time. You'd think that'd be obvious, but it's clear that really smart people don't always think about that.
What changed is that Python 3 was modified, and libraries were added to both Python 2 and 3, so that it started to became practical to write code that could work on both 2 and 3. The result: instead of having to transition your entire program and all its dependencies simultaneously, people could work incrementally, making little improvements here and there so that programs would keep working in 2, and eventually work in 3 as well. Various hacks were also created so that Python 3 could handle stuff like filenames (!).
It would have been much better had the Python 3 developers made it easier to transition from 2 to 3 from the start, but it's finally easier. It's still somewhat painful; mercurial still hasn't transitioned, and it's not clear that they ever will. I'm unhappy about what the Python developers did earlier, but I think they've made good strides in trying to finally make it much easier to transition from Python 2 to Python 3.
The main takeaway here should be, "make it as PAINLESS as POSSIBLE for your existing users to transition to a new version". It's okay to change APIs, but use different names or use versioned APIs. It's okay to change other things, but make it easy for people to modify parts of their program at a time. You'd think that'd be obvious, but it's clear that really smart people don't always think about that.
Other languages rarely have backwards incompatible changes. C or C++ from 20 years ago would still compile perfectly fine with modern compilers.
As far as I know, Go, Java, C#, C++, C and Javascript have never broken backwards compatibility.
Python 3, and Perl 6 did. I don't think anyone uses Perl 6.
As far as I know, Go, Java, C#, C++, C and Javascript have never broken backwards compatibility.
Python 3, and Perl 6 did. I don't think anyone uses Perl 6.
The whole point of Perl 6 was to break compatibility, as the implementation of Perl 5 was deemed good enough to keep existing code running.
In the meantime, Perl 5 keeps getting better while mostly maintaining backwards compatibility. So basically we avoided the problem of transferring people from one language version to the next by allowing them to stay with Perl 5. Perl 5 maintains backwards compatible with code written 30 years ago for the first version of Perl. Which is the reason you have to opt-in to strict mode.
The last major change in Perl 5 that caused widespread problems was done to improve security. (v5.18 hash randomization) Mostly what it did was cause already existing bugs in user code to happen more often, making them easier to track down.
Also thanks for pointing out that not enough people use Perl 6, as it is an awesome language that brings in a lot of ideas from other languages and makes them seem as if they have always belonged together.
In the meantime, Perl 5 keeps getting better while mostly maintaining backwards compatibility. So basically we avoided the problem of transferring people from one language version to the next by allowing them to stay with Perl 5. Perl 5 maintains backwards compatible with code written 30 years ago for the first version of Perl. Which is the reason you have to opt-in to strict mode.
The last major change in Perl 5 that caused widespread problems was done to improve security. (v5.18 hash randomization) Mostly what it did was cause already existing bugs in user code to happen more often, making them easier to track down.
Also thanks for pointing out that not enough people use Perl 6, as it is an awesome language that brings in a lot of ideas from other languages and makes them seem as if they have always belonged together.
I'm on AWS currently. They keep their old / uncool stuff around in a nice way (and totally seem to drop ball on other stuff that matters less like their new Amazon Linux 2 effort - python 3 support in that is totally wonky etc).