The State of Developer Ecosystem in 2018(jetbrains.com)
jetbrains.com
The State of Developer Ecosystem in 2018
https://www.jetbrains.com/research/devecosystem-2018/
61 comments
In quant finance space, Python has been gradually taking over matlab/sas for signal/model implementation.
I'd love to see signs of Julia growing in that space
It is indeed growing as a high-performance computation-focused language, but not to the scale of Python yet.
After a 6 month foray into Ruby/Rails, was just going to hop back into learning Spring Boot today, brush up on Java 8 skills and the like.
Great to see Java is still #1 Hah!
Great to see Java is still #1 Hah!
What's making you leave the Ruby ecosystem?
What's suddenly making Python so popular? I mean, it has been around since the 90s, why is it suddenly gaining tracking in 2017-18? What changed?
I wouldn't call it sudden at all. Python is great for the reason Java is great: diverse robust ecosystem for a ton of different fields. It's about ubiquity.
It's installed on virtually everything already. It's one of the easiest languages to learn (the basics), and is used as a teaching language. Even outside of CS everyone knows it.
Used in web programming (Django, Flask, etc.), numerical computing (Jupyter, numpy, scipy, pandas), AI/ML (TensorFlow, scikit-learn), bioinformatics, easy bindings to EVERYTHING (openCV, bullet physics, kafka, zookeeper, protobuf, json, xml).
It's usually concise enough, mostly straightforward to read, reasonably structured (though big codebases can become a pain). In most companies you see python used in tooling ALL THE TIME. It's fantastic for ducktape programming; often 'promoting' a largeish bash script to python. It ate perl's place there before.
I've even seen ipython work well as a login shell. Jeezz.
Sure the 2.7 -> 3 thing was a ton of drama, the package manager situation changes ever few years, but hey: a lesser language and ecosystem would have fallen apart under it.
Disclaimer: I'm a Java+Python fan boy in all their incarnations.
It's installed on virtually everything already. It's one of the easiest languages to learn (the basics), and is used as a teaching language. Even outside of CS everyone knows it.
Used in web programming (Django, Flask, etc.), numerical computing (Jupyter, numpy, scipy, pandas), AI/ML (TensorFlow, scikit-learn), bioinformatics, easy bindings to EVERYTHING (openCV, bullet physics, kafka, zookeeper, protobuf, json, xml).
It's usually concise enough, mostly straightforward to read, reasonably structured (though big codebases can become a pain). In most companies you see python used in tooling ALL THE TIME. It's fantastic for ducktape programming; often 'promoting' a largeish bash script to python. It ate perl's place there before.
I've even seen ipython work well as a login shell. Jeezz.
Sure the 2.7 -> 3 thing was a ton of drama, the package manager situation changes ever few years, but hey: a lesser language and ecosystem would have fallen apart under it.
Disclaimer: I'm a Java+Python fan boy in all their incarnations.
> I'm a Java+Python fan boy in all their incarnations.
Yet not mention of jython :)
Yet not mention of jython :)
Fan of Jython (though I have a few gripes) excited for graalpython.
If you can get really really good interop you can throw away half of Spring, Guice, dependency injection, etc.
You can use python code as your configuration format. Have it set everything up. Glue all of your Java pieces together for your service. Runtime is and critical paths are all Java.
That's the dream anyhow.
If you can get really really good interop you can throw away half of Spring, Guice, dependency injection, etc.
You can use python code as your configuration format. Have it set everything up. Glue all of your Java pieces together for your service. Runtime is and critical paths are all Java.
That's the dream anyhow.
There'a a fan boy for Java?
Not a replacement for Perl no. It was a great sales trick but as we know a sales trick is not about the truth. But thankfully all the crappy code gets written in Python or JS these days.
> why is it suddenly gaining tracking in 2017-18
For data science and machine learning, it makes sense because the most popular packages are all in Python at this moment (strictly speaking, they have interfaces in Python but the underlying native code is in C++).
For academics who do data analysis and scientific computing, the last few years have seen an explosion of scientific packages available in Python. The major packages, NumPy/SciPy/matplotlib/pandas, also become more feature-complete to the point that one can do everything that is used to be done in MATLAB now in this set of packages. That's free MATLAB with better performance and more potential for optimization. No wonder people are switching to the Python ecosystem.
For data science and machine learning, it makes sense because the most popular packages are all in Python at this moment (strictly speaking, they have interfaces in Python but the underlying native code is in C++).
For academics who do data analysis and scientific computing, the last few years have seen an explosion of scientific packages available in Python. The major packages, NumPy/SciPy/matplotlib/pandas, also become more feature-complete to the point that one can do everything that is used to be done in MATLAB now in this set of packages. That's free MATLAB with better performance and more potential for optimization. No wonder people are switching to the Python ecosystem.
FWIW: For the first ten years of my career, I primarily developed using Java. By about 2008 or so I was tired of looking at SomeAbstractFactoryVistorSingleton messes. I got into a project where I had green-field opportunity to try something different and Python just felt like a breath of fresh air. A few years went by and I gave node.js a try, thinking that Javascript would inevitably win out on the server side as well. Didn't take long to go back to Python - its stable, mature, performant, and syntactically simple. It does what I need it to for now.
In Python we deal with @decoratorDecoratorMetaClassProxy instead.
Entirely true; people can make messes in any language.
Python mostly replaced Ruby(Rails) as the first choice for beginners learning backend for the web.
And Python is becoming the first choice for the Machine Learning and even Data Science.
And Python is becoming the first choice for the Machine Learning and even Data Science.
> Python mostly replaced Ruby(Rails) as the first choice for beginners learning backend for the web.
Is that really the case? I mean I adopted Python and Flask for web work quite awhile ago, but I still see a lot more Ruby web devs around here than anything else.
Is that really the case? I mean I adopted Python and Flask for web work quite awhile ago, but I still see a lot more Ruby web devs around here than anything else.
Geographic region must be important, because I see very few Ruby devs around here compared to PHP and .NET.
Certainly for London, which practically thrives on Ruby and Rails and has done so for the last decade. There hasn't really been a shift - it's basically the case that you start with Rails (or maybe Node) and move on from there, or some times actually go back to Rails because it's so easy to hire for.
I don't see that reflected in the stats for London on Indeed.co.uk. If you search by title there are currently 25% more Django jobs than Rails whilst Python:Ruby is 4.2:1. Node:Rails is about 3:1. I've also noticed a steady decline in Rails job postings. You're seeing about half the number of Rails postings compared with a year ago. Another interesting trend is that Python has more adoption outside London than Ruby in relative terms. Ruby is very much a niche London phenomenon unlike the USA where Ruby/Rails jobs are spread more evenly across the country.
Python has much better support for data anlysis/machine learning, but, honestly, I don't hear about django or other python web frameworks anywhere as much as I hear about rails. Have you got a source for your first point?
As a counterpoint: I never hear about rails being used for anything, except maybe GitHub. I personally also don't know a single person that knows how to write code in Ruby. We both might live in a bubble.
KloudTrader is powered by Rails :D
It was an impression of mine, but it seems it was just bias of availability because I heard some bootcamps were changing Rails for Python.
But I believe I was wrong on this one. A quick search on a page I googled for "best coding bootcamps" led me to this page[0] where a quick ctrl-f showed the results: Ruby = 34 results, Node = 18, Python = 12.
[0] https://www.switchup.org/research/best-coding-bootcamps
But I believe I was wrong on this one. A quick search on a page I googled for "best coding bootcamps" led me to this page[0] where a quick ctrl-f showed the results: Ruby = 34 results, Node = 18, Python = 12.
[0] https://www.switchup.org/research/best-coding-bootcamps
Where is your data on RoR being replaced by Python Seneca? Is it Anecdotal or? I am Actually curious.
As you can see from the Python usage chart most use it for Data Analysis
Data analysis 52% Web development 45%
And most popular libraries :
NumPy / matplotlib / SciPy / Pandas and similar 47% Django 45%
Data analysis 52% Web development 45%
And most popular libraries :
NumPy / matplotlib / SciPy / Pandas and similar 47% Django 45%
I agree with the other commenters about data science and web dev. And as a result, it’s extremely well situated to be the language non-coders reach to when trying to learn to code. I work at a trading firm that has pretty much fully invested in python as a tool to bring preciously non-technical users closer to engineering.
I don't know for sure, but some possible reasons include: 1. I've read that a lot of universities have adopted it as the first language to teach CS students (the local high school also uses it to teach programming to students), 2. When I read questions about what to move to from PHP for a web backend there are lots of recommendations to use Python, 3. It seems like a lot of people use it for data analysis tasks, 4. Python is often used with hardware like Arduino, 5. It is often used for scripting tasks.
Yes, my university switched all 100-level CS papers from Java to Python quite a number of years back (~10). Then came 200-level EE classes, straight into C on bare-metal. :D
Merely anecdotal but: many people interested in machine learning start with Python; many people wanting to do "serverless" things choose Python; many younger devs find Python a nice place to run away to from Java; and all this together means it's pretty easy to find devs who either know Python or are quite happy to learn it; and that in turn makes it a "safe" language choice when you aren't otherwise committed to the features of any particular language. (And running in Lambda can be a pretty big deal.)
It's a virtuous/vicious circle, depending on what you think of Python. But I think "Python is for ML" is the unofficial motivation behind a lot of it, because if you're starting to build your career and Python is an option then for sure you're going to mess around with scikit-learn and put that on your CV.
[edit]: Oh and boto3 probably accounts for a fair amount of Python use these days. I actually prefer the AWS support in golang but boto3 usually "just works" despite some very weird quirks.
It's a virtuous/vicious circle, depending on what you think of Python. But I think "Python is for ML" is the unofficial motivation behind a lot of it, because if you're starting to build your career and Python is an option then for sure you're going to mess around with scikit-learn and put that on your CV.
[edit]: Oh and boto3 probably accounts for a fair amount of Python use these days. I actually prefer the AWS support in golang but boto3 usually "just works" despite some very weird quirks.
I believe it's due to consistently good performance over the years:
* The outstanding documentation site.
* Execution speed for an interpreted language.
* Highly transparent language evolution, managed by the nonprofit Python Foundation.
* Semantic versioning.
* Core developer communication is in English.
* Excellent module definition and import system.
* Culture of explicitness, not implicitness.
The ecosystem has a few huge weaknesses (e.g. packaging and dependency management) which put me off for the most part. But the strengths seem to be overwhelming enough to put Python on a success trajectory.
* The outstanding documentation site.
* Execution speed for an interpreted language.
* Highly transparent language evolution, managed by the nonprofit Python Foundation.
* Semantic versioning.
* Core developer communication is in English.
* Excellent module definition and import system.
* Culture of explicitness, not implicitness.
The ecosystem has a few huge weaknesses (e.g. packaging and dependency management) which put me off for the most part. But the strengths seem to be overwhelming enough to put Python on a success trajectory.
> * Core developer communication is in English.
Just curious, is there any popular programming language whose core developers communicate in a language other than English?
Just curious, is there any popular programming language whose core developers communicate in a language other than English?
Look at the SentDex Python Programming Youtube channel.
The amount of crazy stuff he teaches with python is amazing. No other language can promise that calibre of idea expressing power coupled with ease of development.
The amount of crazy stuff he teaches with python is amazing. No other language can promise that calibre of idea expressing power coupled with ease of development.
I'd argue Ruby and Clojure are an order of magnitude more expressive and much more complete languages. Python has a lot of cruft such as crippled lambdas, __dunder__, self everywhere, add-on regexes, method/function confusion, no :? ternary, quoted string dict keys and no switch/case, to name a few.
To push back on a few of these:
Lambdas are best replaced with `def` in Python - you can define a function anywhere and naming it has advantages. It's a tiny bit more verbose, but in any situation the "crippled" lambda isn't enough, it would probably be less clear in a single expression anyway. I get the arguments that it's less expressive this way, but honestly, it's one of those cases where I think it generally increases code quality.
There is absolutely a ternary operator, it just uses a different order and keywords rather than symbols - `x ? y : z` is `y if x else z` - I think this is much better - it reads much more obviously, uses the keywords from the `if` construct it has the same basic functionality as, and isn't symbol-based, which is always a huge pain point for people working out what the ternary operator is for the first time in other languages. I'd argue this is a place where Python is more expressive than other languages.
Self everywhere is debatable - I get why it's a pain point, but it also does decrease magic and I find new programmers actually pick the object system up quicker thanks to it. I think I wouldn't do it if I was designing a language, but it's not the worst thing ever.
The rest have pros/cons but I'm generally more aligned with, but no language is perfect. I think Python is one of the best, particularly as a first language - a lot of the problems people complain about (including your list) stem from familiarity with other languages that do things another way. Python is more opinionated than most languages in terms of having right ways to do things.
It's interesting, because I like languages on both ends of that spectrum - Scala is so not-opinionated there are a lot of ways to do anything, and Elm is so highly opinionated there is almost never another way to do something.
I think in general though, a language should be opinionated, because consistency and readability are generally more important than writing it. I think Python is exceptionally good at that.
Lambdas are best replaced with `def` in Python - you can define a function anywhere and naming it has advantages. It's a tiny bit more verbose, but in any situation the "crippled" lambda isn't enough, it would probably be less clear in a single expression anyway. I get the arguments that it's less expressive this way, but honestly, it's one of those cases where I think it generally increases code quality.
There is absolutely a ternary operator, it just uses a different order and keywords rather than symbols - `x ? y : z` is `y if x else z` - I think this is much better - it reads much more obviously, uses the keywords from the `if` construct it has the same basic functionality as, and isn't symbol-based, which is always a huge pain point for people working out what the ternary operator is for the first time in other languages. I'd argue this is a place where Python is more expressive than other languages.
Self everywhere is debatable - I get why it's a pain point, but it also does decrease magic and I find new programmers actually pick the object system up quicker thanks to it. I think I wouldn't do it if I was designing a language, but it's not the worst thing ever.
The rest have pros/cons but I'm generally more aligned with, but no language is perfect. I think Python is one of the best, particularly as a first language - a lot of the problems people complain about (including your list) stem from familiarity with other languages that do things another way. Python is more opinionated than most languages in terms of having right ways to do things.
It's interesting, because I like languages on both ends of that spectrum - Scala is so not-opinionated there are a lot of ways to do anything, and Elm is so highly opinionated there is almost never another way to do something.
I think in general though, a language should be opinionated, because consistency and readability are generally more important than writing it. I think Python is exceptionally good at that.
Alternately, `x and y or z`
That's a historic method that was popular before the ternary operator was added, but I'd recommend against it. Primarily because it's just less readable and obvious what the intent is, but also because if `y` is say `[]`, it won't work because that evaluates to `false`.
Deep/Machine Learning
Nothing changed. It had traction for more than a decade.
Machine learning/data science, definitely.
I'm personally not a big fan of Python at all (actually I hate it), but one can't deny its overwhelming dominance in those spaces.
I feel that it's only just starting to grow, and will continue to get bigger.
I'm personally not a big fan of Python at all (actually I hate it), but one can't deny its overwhelming dominance in those spaces.
I feel that it's only just starting to grow, and will continue to get bigger.
you hate it? care to elaborate?
Yes, hate. It boils down to my personal preference, though, so take it with less than a grain of salt.
The MAIN things that bother me: * Slowness * Enforced rules on whitespace * Incompatibility between 2.7 and 3.x * The fact that many tools still use 2.7, making something virtualenv necessary at all * Needing both pip and conda (i.e. to try Caffe) * Extreme dynamic typing means code completions are often poor or of little help * Windows support often lacking
There's actually a _lot_ more, but for the sake of not hijacking this thread, I'm going to stop my list there.
Anyways, those are just my personal preferences; I don't expect those to be reasons for other people to not like Python.
The MAIN things that bother me: * Slowness * Enforced rules on whitespace * Incompatibility between 2.7 and 3.x * The fact that many tools still use 2.7, making something virtualenv necessary at all * Needing both pip and conda (i.e. to try Caffe) * Extreme dynamic typing means code completions are often poor or of little help * Windows support often lacking
There's actually a _lot_ more, but for the sake of not hijacking this thread, I'm going to stop my list there.
Anyways, those are just my personal preferences; I don't expect those to be reasons for other people to not like Python.
I agree with some of your points even though I love Python - but what is it you find lacking about the Windows support? Have you looked at Mark Hammond's pywin32 package and the accompanying book[1]?
The book is written for Python 1.5, but win32 hasn't changed much and neither has the core Python syntax, so it's still very much applicable. (The one major difference is that the book had to concern ifself with converting Unicode strings everywhere, whereas in Python 3 with current pywin32 it just works.)
[1] https://www.amazon.com/Python-Programming-Win32-Windows-Prog...
The book is written for Python 1.5, but win32 hasn't changed much and neither has the core Python syntax, so it's still very much applicable. (The one major difference is that the book had to concern ifself with converting Unicode strings everywhere, whereas in Python 3 with current pywin32 it just works.)
[1] https://www.amazon.com/Python-Programming-Win32-Windows-Prog...
> Slowness
If anyone complains about Python performance and don't use PyPy....they are poking themselves in the eye.
If anyone complains about Python performance and don't use PyPy....they are poking themselves in the eye.
PyPy isn’t compatible with several things I (try to) use, namely TensorFlow.
That’s another thing on the list - I don’t like needing multiple versions of the interpreter installed.
That’s another thing on the list - I don’t like needing multiple versions of the interpreter installed.
Lots of things don't have language bindings with Tensorflow...Lisp, Ruby, JS.
That's true - I'm working on porting bindings for Dart.
... Because I hate Python, haha. Anyways, for the most part, I have to suck it up and just use Python.
... Because I hate Python, haha. Anyways, for the most part, I have to suck it up and just use Python.
AI, ML & data science. Pity Ruby never got off it's ass to compete otherwise the impending Ruby speed boosts might have actually made a difference.
More universities adopting it might be one of many reasons.
Can confirm. My alma mater switched to it as their primary introductory language in about 2006. They had moved from Pascal to C++ to Java, with C++ and Java only lasting a few years each. I got the sense this was a trend in undergraduate academia.
[deleted]
How does javascript have more penetration than HTML/CSS? I can't think of that many reasons to use it if you aren't doing web work.
I take it that you don't use node? I write all my scripts in node instead of bash.