Storm 2.0(storm.apache.org)
storm.apache.org
Storm 2.0
https://storm.apache.org/2019/05/30/storm200-released.html
242 comments
At another point you just accept you'll have to hire people who don't fit the classic job requirement of "X years experience in Lang" and hire people who don't know Lang at all but want to learn. A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding.
On the more meta level it's weird what people expect devs to learn on the job or know up front. Companies expect employees to keep up with the latest nonsense in web dev, but can't pick up a new straightforward language? Unless maybe another Algol like Go or TypeScript? This Steve Yegge quote captures some of it:
"For some reason, programmers love to learn new stuff, as long as it's not syntax."
On the more meta level it's weird what people expect devs to learn on the job or know up front. Companies expect employees to keep up with the latest nonsense in web dev, but can't pick up a new straightforward language? Unless maybe another Algol like Go or TypeScript? This Steve Yegge quote captures some of it:
"For some reason, programmers love to learn new stuff, as long as it's not syntax."
> A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding.
I can attest to this fact, as part of a unicorn upstart with dev offices in Sao Paulo and Berlin. Clojure onboarding has never been a major roadblock for new engineers. And we never hire asking for "X years of lang experience". Almost everyone in the (~300 strong) engineering team started with zero-to-little Clojure background.
I can attest to this fact, as part of a unicorn upstart with dev offices in Sao Paulo and Berlin. Clojure onboarding has never been a major roadblock for new engineers. And we never hire asking for "X years of lang experience". Almost everyone in the (~300 strong) engineering team started with zero-to-little Clojure background.
What do you look for? I mean the "X years of experience" is a shorthand proxy for "can get stuff done in the language we use". So I imagine you must have some other way of getting an idea of how a candidate can get things done. Would love to hear more about it.
What we have found is that it is enough to verify 'can get stuff done', and leave out 'in the language we use'. So, much of the interview turns out to be a process of validating two things:
1. can do what is claimed in the language of choice.
2. gauge enthusiasm and interest to pick up our language.
FWIW, that's been our experience as well (both at a largeish publicly traded company and now at a small private consultancy). Go get work sample tests :)
(It's fine if your work sample test really just focuses on one language! But then make it part of the rubric if that's the answer you want, and tell the applicant about that rubric.)
(It's fine if your work sample test really just focuses on one language! But then make it part of the rubric if that's the answer you want, and tell the applicant about that rubric.)
Thanks!
I can see 1. being pretty easy to suss out (paid take home coding assignment is how I'd do it).
How do you assess 2.?
I can see 1. being pretty easy to suss out (paid take home coding assignment is how I'd do it).
How do you assess 2.?
Mostly in our conversations, in an informal way. I would say the assumption we make is that developers with a penchant for any form of functional prog and/or Lisps, and with an acute focus on testing have an easy time grasping Clojure. And we bias our rubric towards this.
Also, we encourage our candidates to choose Clojure for their take-home assignment, even if they do not have any prior taste of it. This, for those of them who choose that option, gives an idea of if they would enjoy working in it. Our rubric however does not have a bias against developers who do not choose Clojure for their assignments, and we make this explicit to the candidates as well.
Also, we encourage our candidates to choose Clojure for their take-home assignment, even if they do not have any prior taste of it. This, for those of them who choose that option, gives an idea of if they would enjoy working in it. Our rubric however does not have a bias against developers who do not choose Clojure for their assignments, and we make this explicit to the candidates as well.
I guess the difference here is if they have to learn Clojure just to contribute to a single project, then they may be put behind some other things they want to learn and contribute to first.
Yes it seems strange to me that a Java developer is willing to learn Spring, a massive ad-hoc framework which exists in part because of Java's weaknesses, rather than functional programming and Clojure (a lifelong skill that will benefit their Java programming).
This is not the answer that HN wants to hear, but Spring has enormous market share and makes your resume more marketable for tens of thousands of jobs. Clojure, or any other niche language you might insert here, does not help with career building on that level.
For every one shop that looks at darling niche stuff and thinks, "Ooh, that probably makes him or her a better programmer", there's ten shops who look at it and think, "They're probably going to be insufferable to work with, undermine the architects at every turn, and spend their whole time complaining about the tools we use".
As an aside, if you see Spring as existing to correct issues with the underlying Java language, then I'm highly skeptical that you've ever had meaningful experience with either one. Spring is a portfolio of libraries, stretching across over a dozen business application development domains, with inversion of control as the only common thread. NO language has all that baked into its standard library, and you'd have to pull down a bajillion NPM packages (or whatever) to even come close to approximating it.
For every one shop that looks at darling niche stuff and thinks, "Ooh, that probably makes him or her a better programmer", there's ten shops who look at it and think, "They're probably going to be insufferable to work with, undermine the architects at every turn, and spend their whole time complaining about the tools we use".
As an aside, if you see Spring as existing to correct issues with the underlying Java language, then I'm highly skeptical that you've ever had meaningful experience with either one. Spring is a portfolio of libraries, stretching across over a dozen business application development domains, with inversion of control as the only common thread. NO language has all that baked into its standard library, and you'd have to pull down a bajillion NPM packages (or whatever) to even come close to approximating it.
> if you see Spring as existing to correct issues with the underlying Java language, then I'm highly skeptical that you've ever had meaningful experience with either one.
Let's please avoid personal attacks if at all possible. Spring started out with the dependency injection framework and "aspect-oriented programming". It was marketed as an alternative to EJB. Web frameworks and other add-ons came later. If you don't see DI and AspectJ as working around Java's weaknesses, then I encourage you to try a more expressive language.
In an expressive language (e.g. Clojure or Haskell), DI and AspectJ could be, for example, replaced by reifying a program as data and using modular interpreters. One interpreter using mock data, another talks to the database, uses transaction boundaries and logs. This sort of solution is unfortunately just not very practical in Java.
Let's please avoid personal attacks if at all possible. Spring started out with the dependency injection framework and "aspect-oriented programming". It was marketed as an alternative to EJB. Web frameworks and other add-ons came later. If you don't see DI and AspectJ as working around Java's weaknesses, then I encourage you to try a more expressive language.
In an expressive language (e.g. Clojure or Haskell), DI and AspectJ could be, for example, replaced by reifying a program as data and using modular interpreters. One interpreter using mock data, another talks to the database, uses transaction boundaries and logs. This sort of solution is unfortunately just not very practical in Java.
DI is just another name for "passing values to functions". We did it in Java just fine before Spring became so popular and Spring doesn't add that much for this particular way of writing Java code. Of course Spring is nice as a whole, even if huge and much more than DI, but the benefits of using it just to specify lazy proxy singletons using annotations are overblown. Autowiring becomes seductive so you get situations when you still need to test and some devs now have a component class with 10 private autowired dependencies and ask questions on how to test it that lead to in order of awfulness using reflection in the test to bypass 'private' and assign dummy values to the dependencies, using Spring's JUnit test runner and mock beans, rewriting the class a bit to use constructor or setter-method autowires that you can set without bringing in all the spring machinery, or break up the test so it depends on less and individual parts may not even need to be Spring components at all.
In another expressive language like Common Lisp, it has generic functions which 1) let you dispatch on more than just the class type of the first argument (implicitly in Java, explicitly 'self' in Python) which is a feature that by itself removes a lot of Java back-bending and 2) you can specify :before and :after and :around methods to provide the benefits of aspects with the built-in function compute-applicable-methods available to help see everything that might apply when invoking a method with specific arguments. There's also metaclasses which open other doors. There's just a whole lot of built-in expressiveness in the languages beyond the Java and JavaScript ecosystems.
In another expressive language like Common Lisp, it has generic functions which 1) let you dispatch on more than just the class type of the first argument (implicitly in Java, explicitly 'self' in Python) which is a feature that by itself removes a lot of Java back-bending and 2) you can specify :before and :after and :around methods to provide the benefits of aspects with the built-in function compute-applicable-methods available to help see everything that might apply when invoking a method with specific arguments. There's also metaclasses which open other doors. There's just a whole lot of built-in expressiveness in the languages beyond the Java and JavaScript ecosystems.
Software developers' skills are mostly fungible– supply meets demand.
I've been happily employed (and regularly solicited) as a Clojure developer for over 8 years.
There's no shortage of Clojure dev positions for developers that want to write Clojure.
I've been happily employed (and regularly solicited) as a Clojure developer for over 8 years.
There's no shortage of Clojure dev positions for developers that want to write Clojure.
I've worked with plenty of devs in my time who don't really understand what they're doing and just copy and paste code from Stack Overflow until it seems to work. A lot of the enterprise takes the monkeys with typewriters approach to software development.
I think there are cultural reasons for that because enterprise treats devs like fungible cogs that you replace when they burn out. It tends to be highly dehumanizing environment where devs have no say, no autonomy, and no ownership. This naturally pushes out people who have options, and you end up with a pool of devs who just want any job that pays the bills.
I think there are cultural reasons for that because enterprise treats devs like fungible cogs that you replace when they burn out. It tends to be highly dehumanizing environment where devs have no say, no autonomy, and no ownership. This naturally pushes out people who have options, and you end up with a pool of devs who just want any job that pays the bills.
>>I think there are cultural reasons for that because enterprise treats devs like fungible cogs that you replace when they burn out.
Its mostly approach to any work in general. If it were upto them, these are really the kind of the people who would dig earth with spoons and shovels, instead of heavy machinery.
The idea is simple, should you use Spring + Java, you are likely to need say 60+ people to run a product well. It could take 15 to make it work in Clojure. But having 60 people has its own advantages from their perspective. Say 10 people decide to leave, they can hire 10 replacements for the lowest prices from the market, train them on the application and get it going. Eventually the whole team could leave and get replaced this way, reducing the problem to a bit like the Ship of Theseus paradox. But it works for them.
If they use Clojure, they have to treat people well, pay them well to retain them. Because there teams tend to be small and a exodus puts everything at risk.
They have to use spoons and shovels, personnel trained to work on heavy machinery are expensive, training is expensive and replacing them is expensive.
Its mostly approach to any work in general. If it were upto them, these are really the kind of the people who would dig earth with spoons and shovels, instead of heavy machinery.
The idea is simple, should you use Spring + Java, you are likely to need say 60+ people to run a product well. It could take 15 to make it work in Clojure. But having 60 people has its own advantages from their perspective. Say 10 people decide to leave, they can hire 10 replacements for the lowest prices from the market, train them on the application and get it going. Eventually the whole team could leave and get replaced this way, reducing the problem to a bit like the Ship of Theseus paradox. But it works for them.
If they use Clojure, they have to treat people well, pay them well to retain them. Because there teams tend to be small and a exodus puts everything at risk.
They have to use spoons and shovels, personnel trained to work on heavy machinery are expensive, training is expensive and replacing them is expensive.
I think it's interesting that people (willingly?) pigeonhole themselves to the extent that they're a "Java developer" or a ".NET Engineer" rather than more simply a "Software Engineer".
I'm a Software Engineer, and while I have my go-to tools, I'd like to think I make an informed choice to use what fits the task at hand .
An Engineer who's a comfortable polygot is perhaps also a better Engineer overall.
I'm a Software Engineer, and while I have my go-to tools, I'd like to think I make an informed choice to use what fits the task at hand .
An Engineer who's a comfortable polygot is perhaps also a better Engineer overall.
I do the same, but I also happen to acknowledge that I slowly transition into X Developer, depending on which project I am on.
It is the usual "Jack of all trades, master of none." consulting, but it does come in handy when we need to deal with mixed stacks end-to-end.
It is the usual "Jack of all trades, master of none." consulting, but it does come in handy when we need to deal with mixed stacks end-to-end.
I'm willingly pigeonholing myself as a "software developer mostly working on the JVM" just to be honest about my (current) main strengths/experince.
Throwing me into a Go/Python/whatever project would first need some time for me to get up to speed with the language.
I think it's a vicious circle: management used year of experienc with <tool/language> as a hiring filter, so devs focus on building years of experience with in-demand tools/languages to maintain maximum career opportunity, and the laser-like focus developers exhibit reinforces the idea that devs can't be flexible on tools and languages, supporting the use of the classic hiring filter.
Also, to get there, you're already accepting the assertion that the move isn't a management decision :-)
Spring is so large that a lot of job offerings require X years of Spring experience.
Benefit their programming maybe but I don't see who Clojure will benefit anyone's Java as the two languages are poles apart.
They both share the underlying JVM library ecosystem, and as Java increasingly applies functional programming to its mainstream frameworks, clojure is a good illustration of how a near-pure functional style can work within the JVM environment
Well, this isn't a hiring situation, this is a getting contributors to an open-source project. When you're looking for volunteers, the tolerable amount of friction is often less than if you're paying someone for their time.
Yes, I was responding mostly to the grandparent's move towards discussing this sort of tradeoff in the commercial space, not just in the open source space. To stick with open source, it could become the case that moving away from Clojure leads to a cooling of contributors, existing or new. If you're looking for an Apache project to start contributing to, there's no shortage of Java projects. But if you like Clojure or want to learn it better, hey look there used to be this Clojure one, cool. If you liked the Storm project specifically and wanted to conribute, I don't think the language would get in the way that much. (Speaking personally if I had more time and energy I'd like to try and contribute to the Godot project despite my desire to never have to program in C++ again. It's a sufficiently cool project to overcome that.)
Optimizing for new volunteer contributors goes beyond core language choice anyway. For an interesting description of how the game Battle for Wesnoth tried to do it, see: http://aosabook.org/en/wesnoth.html
Optimizing for new volunteer contributors goes beyond core language choice anyway. For an interesting description of how the game Battle for Wesnoth tried to do it, see: http://aosabook.org/en/wesnoth.html
My team has been using Clojure for close to a decade, and we found the opposite to be the case. While the pool of applicants is smaller, so is the noise ratio. Clojure being niche means that you get people who are willing to look outside the mainstream, and are typically genuinely interested in programming.
We also rarely hire devs who already know Clojure, and we typically just train on the job. It's really not hard to do when you already have people who know the language on the team. In fact, if somebody can't learn Clojure, I would question if they can learn to work on a large project in general. Dev practices, architecture, code styles, tooling, and so on tends to change quite a bit from company to company. The language is only small part of that.
In case of Storm, Apache commons is run by Java devs who have zero interest in learning Clojure. So, it's not surprising they would rewrite Storm in their preferred language.
We also rarely hire devs who already know Clojure, and we typically just train on the job. It's really not hard to do when you already have people who know the language on the team. In fact, if somebody can't learn Clojure, I would question if they can learn to work on a large project in general. Dev practices, architecture, code styles, tooling, and so on tends to change quite a bit from company to company. The language is only small part of that.
In case of Storm, Apache commons is run by Java devs who have zero interest in learning Clojure. So, it's not surprising they would rewrite Storm in their preferred language.
Same (sure; anecdotal) experience here. Hiring for Clojure has been so much easier for us than hiring for JavaScript.
Granted, these are two different products, and maybe the Clojure one is just more attractive to the right kind of person. Nevetheless, in terms of quality vs. quantity of applicants, Clojure has been massively winning.
Granted, these are two different products, and maybe the Clojure one is just more attractive to the right kind of person. Nevetheless, in terms of quality vs. quantity of applicants, Clojure has been massively winning.
This is salient too though:
The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
It's not totally clear if this is because it's been rewritten Java, but the intrinsic qualities of the language do matter; there are real tradeoffs between maintainability and dynamism/flexibility. I feel like this sometimes gets shortchanged in these narratives.
The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
It's not totally clear if this is because it's been rewritten Java, but the intrinsic qualities of the language do matter; there are real tradeoffs between maintainability and dynamism/flexibility. I feel like this sometimes gets shortchanged in these narratives.
Idiomatic java is like idiomatic C: pretty fast. Most smaller languages prioritize other things. Clojure has immutability and Haskell has purity.
I have noticed this with every "X produces code faster than C": you begin with two programs that use a suboptimal algorithm, then you take your non-c language and try to write C in it. The result is always awful and removes most reasons not to use C in the first place.
This has somewhat changed with rust and in some sense C++,but for other languages my point still stands. They have a nice idiomatic golden path that is fast enough for most cases. Once you need performance badly enough you have to treat you language as an assembler, and then you will always lose to languages that actually are good at that.
I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.
I have noticed this with every "X produces code faster than C": you begin with two programs that use a suboptimal algorithm, then you take your non-c language and try to write C in it. The result is always awful and removes most reasons not to use C in the first place.
This has somewhat changed with rust and in some sense C++,but for other languages my point still stands. They have a nice idiomatic golden path that is fast enough for most cases. Once you need performance badly enough you have to treat you language as an assembler, and then you will always lose to languages that actually are good at that.
I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.
I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.
The same with Clojure, suddenly you start writing Java with parenthesis.
In the JVM, if you want absolute control over performance its better to just write Java for those parts (don't know much about Kotlin).
The same with Clojure, suddenly you start writing Java with parenthesis.
In the JVM, if you want absolute control over performance its better to just write Java for those parts (don't know much about Kotlin).
Kotlin generally is not slower than Java. One exception is that it inserts a lot of null checks. And there are features that will make it even faster than Java, I'm talking about inline lambdas.
The compiler is significantly slower than Java's, though.
That's the sad truth for a lot of modern compiled languages (I think Go is an exception). While I did not dig into their compiler internals, it just seems an inevitable consequence from a powerful language (Go is pretty simple in that regard). C++, Rust, Scala, Kotlin, Swift, they all have significantly longer compilation times compared to their predecessors. Probably that's the price we have to pay.
Chez compiles a 80Kloc project of mine in less than a second, and apparently that is still 2x slower than before they switched to a nanopass compiler.
There is no reason compiling things should not compile at similar speeds to chez unless you are telling your compiler to ootimise everything.
C++/ruae are just exceptionally slow and it became the new bottom line somehow. I know rust does a lot of housekeeping, but after using chez and sbcl, 45s for a 1.5kloc project is killing me inside.
There is no reason compiling things should not compile at similar speeds to chez unless you are telling your compiler to ootimise everything.
C++/ruae are just exceptionally slow and it became the new bottom line somehow. I know rust does a lot of housekeeping, but after using chez and sbcl, 45s for a 1.5kloc project is killing me inside.
Not really.
.NET Native, D, Delphi, Ada and Eiffel also have pretty fast compilers, while offering quite powerful languages.
Lets even pick Turbo Pascal 7.0 for MS-DOS, which was quite feature rich, and was able to compile very large programs in a couple of seconds.
Also Visual C++ with modules preview support, incremental compilation and linking is also reasonably faster than the FOSS alternatives.
It is only a matter how much money one is willing to invest into improving tooling support.
.NET Native, D, Delphi, Ada and Eiffel also have pretty fast compilers, while offering quite powerful languages.
Lets even pick Turbo Pascal 7.0 for MS-DOS, which was quite feature rich, and was able to compile very large programs in a couple of seconds.
Also Visual C++ with modules preview support, incremental compilation and linking is also reasonably faster than the FOSS alternatives.
It is only a matter how much money one is willing to invest into improving tooling support.
As far as I know Google is supporting JetBrains in making the Kotlin compiler faster.
The announcement says they have a new architecture. At the very least it's not a direct performance comparison to the same architecture implemented in Clojure.
> Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.
> Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.
Clojure is marketed as a very fast language (bunch of clever optimisation gets done because data structures are immutable). If the gains were from switching to Java they'd probably have said so less ambiguously.
Surely what they mean is they've rewritten it to be faster, and switched to Java at the same time because they want to open up the contributor pool.
Surely what they mean is they've rewritten it to be faster, and switched to Java at the same time because they want to open up the contributor pool.
BTW, if anyone's looking for another language that stresses immutability on the JVM, you might want to check out Frege, a Haskell-like language with some "tweaks" that make it work well in that environment. https://github.com/Frege/frege
And as a sibling comment mentions, the announcement does clearly state that a rather extensive rewrite was done, including switching to a new architecture, with likely performance improvements coming from that - it's nowhere close to a pure Java vs. Clojure comparison.
And as a sibling comment mentions, the announcement does clearly state that a rather extensive rewrite was done, including switching to a new architecture, with likely performance improvements coming from that - it's nowhere close to a pure Java vs. Clojure comparison.
My impression is that Eta is a better choice, if only because it is compatible with a wider variety of Haskell libraries.
https://eta-lang.org/
https://eta-lang.org/
Eta is a better choice, if you care about Haskell compatibility in the strictest sense. Frege doesn't really pursue this, but the flip side is that it optimizes for working as smoothly as possible with the JVM ecosystem.
I did find that bit interesting. It seems it's missing a leading "due to the commiters' better familiarity with Java."
Since you can argue knowledge of a language does help with maintenance and extensibility. With this, it also makes more sense for the following bit about finding people able to contribute. No contributors is pretty bad for maintenance.
At my work, we use Clojure, but I do often wonder what would happen if the current Clojure devs left. New devs unfamiliar with Lisps, FP, and Clojure would most likely have a hard time maintaining the code base and extending it, and if they try to rush it before really learning the language, would quickly degrade the quality of the code base, compounding the effect.
I think as long as you continue to have one strong Clojure dev on the projects, you'll be fine, as they can direct newcomers and help them transition to Clojure, but if you lost that, I think a rewrite in a different language would make sense, or there's a risk of the code base degrading quite quickly.
Since you can argue knowledge of a language does help with maintenance and extensibility. With this, it also makes more sense for the following bit about finding people able to contribute. No contributors is pretty bad for maintenance.
At my work, we use Clojure, but I do often wonder what would happen if the current Clojure devs left. New devs unfamiliar with Lisps, FP, and Clojure would most likely have a hard time maintaining the code base and extending it, and if they try to rush it before really learning the language, would quickly degrade the quality of the code base, compounding the effect.
I think as long as you continue to have one strong Clojure dev on the projects, you'll be fine, as they can direct newcomers and help them transition to Clojure, but if you lost that, I think a rewrite in a different language would make sense, or there's a risk of the code base degrading quite quickly.
It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much.
Another view of this would be: progress in this industry is made by those who succeed while sticking to their guns and not optimizing their infrastructure for the available worker pool.
Another view of this would be: progress in this industry is made by those who succeed while sticking to their guns and not optimizing their infrastructure for the available worker pool.
You could be right. But I think there are good reasons to believe that there is more to it than that.
I have a strong suspicion that people smart enough to do advanced meta programming on their own are not smart enough to collaborate with equally skilled coworkers on code that uses all those abstractions to the max. Even collaborating with your own former self is more difficult that writing new code.
So I think _some_ dumbing down is inevitable. The question is how that can be enforced. Choosing a dumber language to bludgeon everyone into submission is the nuclear option so to speak.
I often think of this Dijkstra quote about abstraction: "The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise".
But I think there's something missing in this thought. Abstraction, especially when its precise, is also a form of compression. To understand what a particular abstract expression means in a concrete case requires decompression, which may require a lot of mental effort.
These are admittedly extremely half-baked thoughts...
I have a strong suspicion that people smart enough to do advanced meta programming on their own are not smart enough to collaborate with equally skilled coworkers on code that uses all those abstractions to the max. Even collaborating with your own former self is more difficult that writing new code.
So I think _some_ dumbing down is inevitable. The question is how that can be enforced. Choosing a dumber language to bludgeon everyone into submission is the nuclear option so to speak.
I often think of this Dijkstra quote about abstraction: "The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise".
But I think there's something missing in this thought. Abstraction, especially when its precise, is also a form of compression. To understand what a particular abstract expression means in a concrete case requires decompression, which may require a lot of mental effort.
These are admittedly extremely half-baked thoughts...
>I have a strong suspicion that people smart enough to do advanced meta programming on their own are not smart enough to collaborate with equally skilled coworkers on code that uses all those abstractions to the max. Even collaborating with your own former self is more difficult that writing new code.
I'm an unabashed Clojure fanboy and will talk the language up to anyone who will listen, but one thing that I've been thinking about lately is that good judgment on abstraction use is necessary for using such powerful tools. I don't think the place to tackle it is by being "smart enough to understand," but rather by being "disciplined enough to not abstract sometimes." The exciting thing is that discipline doesn't require you to be super smart, just to think it's worth caring about.
Also half-baked, but I think it's something.
I'm an unabashed Clojure fanboy and will talk the language up to anyone who will listen, but one thing that I've been thinking about lately is that good judgment on abstraction use is necessary for using such powerful tools. I don't think the place to tackle it is by being "smart enough to understand," but rather by being "disciplined enough to not abstract sometimes." The exciting thing is that discipline doesn't require you to be super smart, just to think it's worth caring about.
Also half-baked, but I think it's something.
The easy solution to this is to create a culture of using powerful tools, but requiring REALLY good reasons to go far to their edges in terms of metaprogramming.
It's not that much harder to enforce than it is to enforce conventions in less powerful languages, and the convenience of working in a well-designed language is so nice.
It's not that much harder to enforce than it is to enforce conventions in less powerful languages, and the convenience of working in a well-designed language is so nice.
Java is abstraction over JVM which is abstraction over machine code. But I don't need to know anything about JVM bytecode or machine code to write typical Java code. Good abstraction does not require decompression. You're not going down to assembly to understand some Java code. You think on Java code level. That is good abstraction. If you need to decompress what you're reading, then it's a leaky abstraction. It's okay to use leaky abstractions to reduce repetitions, for example. But ideally your abstractions should not require one to learn its implementations in typical cases. For example if your function name is good enough, reader could trust you to do the proper thing and not read its implementation.
>Good abstraction does not require decompression
I think there's a misunderstanding. What I'm calling decompression has nothing to do with digging into the implementation of a particular abstraction. What I mean is merely tracking down all the indirections in the public interface and breaking them down to their concrete meaning in a specific case.
Some languages allow for a lot more moving parts than others. For instance, in Java obj.otherName means that otherName is an instance variable (or theoretically a class variable) and access time is guaranteed to be constant (tbd: caveats). It's not redefinable as it is in C#, Python or Swift. So there's one less thing to track down but also less flexibility.
Or take an extreme example like Go's range loop. It's defined only for a handful of builtin data structures. When you see a range loop, you know what it does without following any further indirections. It can be extremely annoying and create a lot of friction when you have to work with custom data structures. But it can make other people's code easier to read than in other languages.
Or take something like this:
So transforming an expression's possible meanings into its concrete meaning is what I call decompression. There may not be a need for looking anything other than public interfaces.
Also, you're talking about abstractions purely from the perspective of users. But we are often creators and maintainers of abstractions as well when we model a particular problem.
I think there's a misunderstanding. What I'm calling decompression has nothing to do with digging into the implementation of a particular abstraction. What I mean is merely tracking down all the indirections in the public interface and breaking them down to their concrete meaning in a specific case.
Some languages allow for a lot more moving parts than others. For instance, in Java obj.otherName means that otherName is an instance variable (or theoretically a class variable) and access time is guaranteed to be constant (tbd: caveats). It's not redefinable as it is in C#, Python or Swift. So there's one less thing to track down but also less flexibility.
Or take an extreme example like Go's range loop. It's defined only for a handful of builtin data structures. When you see a range loop, you know what it does without following any further indirections. It can be extremely annoying and create a lot of friction when you have to work with custom data structures. But it can make other people's code easier to read than in other languages.
Or take something like this:
a == b
So transforming an expression's possible meanings into its concrete meaning is what I call decompression. There may not be a need for looking anything other than public interfaces.
Also, you're talking about abstractions purely from the perspective of users. But we are often creators and maintainers of abstractions as well when we model a particular problem.
I think the better word you're looking for might be "reify" rather than "decompress". But there is something here that I can relate to. A long time ago I also made a comparison between various languages of "what do you have to understand to fully understand a Hello World program?" An interesting followup to that would be how much you have to understand on top of that to understand a more useful program.
You might be interested in this book, Patterns of Software, which brings up the issue that inheritance in OOP isn't about reuse but about compression and how the implications of that are why it's not had that much success in following through on the "reuse" promise. http://dreamsongs.com/Files/PatternsOfSoftware.pdf
You might be interested in this book, Patterns of Software, which brings up the issue that inheritance in OOP isn't about reuse but about compression and how the implications of that are why it's not had that much success in following through on the "reuse" promise. http://dreamsongs.com/Files/PatternsOfSoftware.pdf
Abstractions are a spectrum. There's always a degree of benefit to be had from peeking under the hood. I'd argue that it's rare that you find an abstraction that can be used well without having to understand, or "decompress", some of it.
Consider JVM and Java: that you can write Java seemingly without any knowledge about JVM is an illusion - you already learned a lot about how things like JVM work under the hood when first learning programming[0]. Learning more about JVM itself lets you write more efficient code and debug better.
Consider functions: it's near-impossible to contain all the information necessary to use a non-trivial function in its header[1]. That's why good code contains comments and other forms of documentation describing the abstraction in more details. Even then, it's not always enough - sometimes it's really much easier to understand what you need by reading the source[2].
Consider any appliance - be it a car, or a radio, or a dishwasher. You can use it to it interface, to some extent at least. But knowing what's going on under the literal hood really does help with use, and especially helps when something goes wrong. If you don't know what's hidden under the abstraction layer, any failure will likely be incomprehensible to you and leave you helpless.
--
[0] - In theory, it might be possible to learn some programming without learning anything about hardware or hardware-emulating abstractions. In reality, I've never seen it or heard of it, and I suspect that the simplest mental model of code execution is isomorphic to somewhat simplified computer or virtual machine.
[1] - Function name + name of its arguments + types of its arguments and return value, if available.
[2] - Then again, implementation code may not capture the entire abstraction either. That's why good code often features comments inside the implementation, explaining the rationale behind some of the less obvious code parts.
Consider JVM and Java: that you can write Java seemingly without any knowledge about JVM is an illusion - you already learned a lot about how things like JVM work under the hood when first learning programming[0]. Learning more about JVM itself lets you write more efficient code and debug better.
Consider functions: it's near-impossible to contain all the information necessary to use a non-trivial function in its header[1]. That's why good code contains comments and other forms of documentation describing the abstraction in more details. Even then, it's not always enough - sometimes it's really much easier to understand what you need by reading the source[2].
Consider any appliance - be it a car, or a radio, or a dishwasher. You can use it to it interface, to some extent at least. But knowing what's going on under the literal hood really does help with use, and especially helps when something goes wrong. If you don't know what's hidden under the abstraction layer, any failure will likely be incomprehensible to you and leave you helpless.
--
[0] - In theory, it might be possible to learn some programming without learning anything about hardware or hardware-emulating abstractions. In reality, I've never seen it or heard of it, and I suspect that the simplest mental model of code execution is isomorphic to somewhat simplified computer or virtual machine.
[1] - Function name + name of its arguments + types of its arguments and return value, if available.
[2] - Then again, implementation code may not capture the entire abstraction either. That's why good code often features comments inside the implementation, explaining the rationale behind some of the less obvious code parts.
You just explained cornerstone of good design
Because no large effect of programming language choice has been observed, and because no theory explains why there would even be one[1], speculating on the causes of a non-existent phenomenon is not very helpful.
[1]: Once a certain point has been reached; theory does predict diminishing returns.
[1]: Once a certain point has been reached; theory does predict diminishing returns.
It's near impossible to objectively measure anything when it comes to programming as projects are neither repeatable nor comparable and test subjects are never blank slates.
I have read quite a few studies, and I always come away thinking that the methods used are inadequate to support any conclusions at all.
But does that mean there is no causal effect of programming language choice? I don't know and I'm not making any claims either way.
I was talking about abstraction in one person's mind vs abstraction shared between a group a people. And what I said could actually be the reason why the choice of programming language has little effect (if that is the case)
I have read quite a few studies, and I always come away thinking that the methods used are inadequate to support any conclusions at all.
But does that mean there is no causal effect of programming language choice? I don't know and I'm not making any claims either way.
I was talking about abstraction in one person's mind vs abstraction shared between a group a people. And what I said could actually be the reason why the choice of programming language has little effect (if that is the case)
> But does that mean there is no causal effect of programming language choice? I don't know and I'm not making any claims either way.
I'm also not making definitive claims, but we can't ignore observations, either. While small effects can appear or disappear depending on methodology, big effects are easy to find and hard to hide, especially in an environment with strong selection pressures. Therefore, the most likely explanation to why no big effect has been detected -- either in studies or in industry -- is that there isn't one, and that should at least be enough to make it the working hypothesis. So before explaining effects that have yet to be detected we should establish that they exist at all, something we have so far failed to do.
There may well be a small effect, and there could still be a large effect, although the chance of that is small unless we have a really good explanation to why we haven't found it.
I'm also not making definitive claims, but we can't ignore observations, either. While small effects can appear or disappear depending on methodology, big effects are easy to find and hard to hide, especially in an environment with strong selection pressures. Therefore, the most likely explanation to why no big effect has been detected -- either in studies or in industry -- is that there isn't one, and that should at least be enough to make it the working hypothesis. So before explaining effects that have yet to be detected we should establish that they exist at all, something we have so far failed to do.
There may well be a small effect, and there could still be a large effect, although the chance of that is small unless we have a really good explanation to why we haven't found it.
Here are some reasons to doubt this working hypothesis:
1) It is inconsistent with the personal experience of most developers. Few would claim that C is just as productive and safe as some garbage collected language (provided that both are suitable for the task at hand), or that a SQL group by statement is not more productive to write than the equivalent procedural code.
2) The methods used to study the subject are completely unfit for purpose.
3) It is inconsistent with the few results that seem at least somewhat well supported because they are simple enough to measure, such as the constant bug count per line of code.
1) It is inconsistent with the personal experience of most developers. Few would claim that C is just as productive and safe as some garbage collected language (provided that both are suitable for the task at hand), or that a SQL group by statement is not more productive to write than the equivalent procedural code.
2) The methods used to study the subject are completely unfit for purpose.
3) It is inconsistent with the few results that seem at least somewhat well supported because they are simple enough to measure, such as the constant bug count per line of code.
> It is inconsistent with the personal experience of most developers.
I disagree. I've been programming for about 30 years now, and most reports I've received -- both from programmers and managers -- are that language doesn't matter; certainly not much. Carefully selected personal "feeling" report are worthless (especially in biased forums), as that would produce even more evidence that homeopathy is effective than that programming languages make a difference.
> Few would claim that C is just as productive and safe as some garbage collected language
I agree, but that's where the many caveats come in. First, my argument is more one of diminishing returns (following Brooks's theory). I.e., languages make an increasingly small difference. C may have been a big improvement over assemby, and Java may have been a smaller but still quite big improvement over C, but now differences are quite small.
Indeed, the effect of GC and memory safety vs C was one that was almost immediately detected by industry (once mature, performant, etc.) and triggered a huge shift. We do not see such shifts now. Companies move among mainstream languages, among non-mainstream languages, and between the two classes in a way that does not suggest a big effect at all.
> or that a SQL group by statement is not more productive to write than the equivalent procedural code.
That's a completely different matter. I don't think anyone believes that writing a 1MLOC program in SQL is easier than writing a similar one in Java or Python, even if the Python/Java one is 1MLOC, and the SQL one is 200KLOC. When the program is small, there can be large differences, but that's a completely different problem (and also explained by theory).
> The methods used to study the subject are completely unfit for purpose.
I disagree, but it doesn't matter. Big effects are easy to find and hard to hide. And in an environment with strong selective pressures, they're found even when no studies at all are conducted.
> It is inconsistent with the few results that seem at least somewhat well supported because they are simple enough to measure, such as the constant bug count per line of code.
It is not. First, the difference in LOC is not as big as PL fans claim (it can be big for small programs, not large ones). Second, you may want to review those results.
I disagree. I've been programming for about 30 years now, and most reports I've received -- both from programmers and managers -- are that language doesn't matter; certainly not much. Carefully selected personal "feeling" report are worthless (especially in biased forums), as that would produce even more evidence that homeopathy is effective than that programming languages make a difference.
> Few would claim that C is just as productive and safe as some garbage collected language
I agree, but that's where the many caveats come in. First, my argument is more one of diminishing returns (following Brooks's theory). I.e., languages make an increasingly small difference. C may have been a big improvement over assemby, and Java may have been a smaller but still quite big improvement over C, but now differences are quite small.
Indeed, the effect of GC and memory safety vs C was one that was almost immediately detected by industry (once mature, performant, etc.) and triggered a huge shift. We do not see such shifts now. Companies move among mainstream languages, among non-mainstream languages, and between the two classes in a way that does not suggest a big effect at all.
> or that a SQL group by statement is not more productive to write than the equivalent procedural code.
That's a completely different matter. I don't think anyone believes that writing a 1MLOC program in SQL is easier than writing a similar one in Java or Python, even if the Python/Java one is 1MLOC, and the SQL one is 200KLOC. When the program is small, there can be large differences, but that's a completely different problem (and also explained by theory).
> The methods used to study the subject are completely unfit for purpose.
I disagree, but it doesn't matter. Big effects are easy to find and hard to hide. And in an environment with strong selective pressures, they're found even when no studies at all are conducted.
> It is inconsistent with the few results that seem at least somewhat well supported because they are simple enough to measure, such as the constant bug count per line of code.
It is not. First, the difference in LOC is not as big as PL fans claim (it can be big for small programs, not large ones). Second, you may want to review those results.
>I agree, but that's where the many caveats come in. First, my argument is more one of diminishing returns (following Brooks's theory). I.e., languages make an increasingly small difference. C may have been a big improvement over assemby, and Java may have been a smaller but still quite big improvement over C, but now differences are quite small.
I don't necessarily disagree, but I'm a bit skeptical about the way in which you use the concept of diminishing returns. If I'm not mistaken then Brooks used diminishing returns in relation to headcount. And this is in fact how economists use the term as well - adding quantitatively more of one production factor.
But that doesn't apply to qualitative changes like introducing new production methods and tools, which is what programming languages are. New programming languages are not simply "more programming language".
If we agree that there have been observable effects of new programming languages in the past, then we cannot exclude the possibility of the same happening in the future.
>First, the difference in LOC is not as big as PL fans claim
What PL fans claim is a bit of a fluffy benchmark, but there are undoubtedly significant differences in LOC, certainly significant enough to make an economic difference.
>I've been programming for about 30 years now...
Shockingly, me too :)
I don't necessarily disagree, but I'm a bit skeptical about the way in which you use the concept of diminishing returns. If I'm not mistaken then Brooks used diminishing returns in relation to headcount. And this is in fact how economists use the term as well - adding quantitatively more of one production factor.
But that doesn't apply to qualitative changes like introducing new production methods and tools, which is what programming languages are. New programming languages are not simply "more programming language".
If we agree that there have been observable effects of new programming languages in the past, then we cannot exclude the possibility of the same happening in the future.
>First, the difference in LOC is not as big as PL fans claim
What PL fans claim is a bit of a fluffy benchmark, but there are undoubtedly significant differences in LOC, certainly significant enough to make an economic difference.
>I've been programming for about 30 years now...
Shockingly, me too :)
> If I'm not mistaken then Brooks used diminishing returns in relation to headcount.
Oh, I'm referring to "No Silver Bullet" and diminishing returns in the sense that languages at best help with "accidental complexity", so the less of it there is, the less languages can help.
> New programming languages are not simply "more programming language".
But there are inflexible theoretical limitations on their utility (even without Brooks). We know the effect of different programming constructs on expressive compression and reasoning costs. The closer we get to the limit, the less benefit we can have.
> then we cannot exclude the possibility of the same happening in the future.
I'm mostly pointing out that it's not happening at present. I'm less sanguine about future advances because of various limitations (and, BTW, Brooks's predictions were called overly pessimistic by PL fans at the time and it turned out they were too optimistic), but I won't rule out another major breakthrough, or possibly two.
> but there are undoubtedly significant differences in LOC, certainly significant enough to make an economic difference.
I don't think I agree with your first assertion (although that depends on what we mean by "significant" here), and I certainly disagree with your second. We simply have not been able to detect or induce such an effect. Here, too, there are theoretical results showing that increased expressiveness cannot lead to cheaper reasoning, which is not an obvious result even in the worst case (because there are far fewer "compressible" programs than non compressible ones).
Oh, I'm referring to "No Silver Bullet" and diminishing returns in the sense that languages at best help with "accidental complexity", so the less of it there is, the less languages can help.
> New programming languages are not simply "more programming language".
But there are inflexible theoretical limitations on their utility (even without Brooks). We know the effect of different programming constructs on expressive compression and reasoning costs. The closer we get to the limit, the less benefit we can have.
> then we cannot exclude the possibility of the same happening in the future.
I'm mostly pointing out that it's not happening at present. I'm less sanguine about future advances because of various limitations (and, BTW, Brooks's predictions were called overly pessimistic by PL fans at the time and it turned out they were too optimistic), but I won't rule out another major breakthrough, or possibly two.
> but there are undoubtedly significant differences in LOC, certainly significant enough to make an economic difference.
I don't think I agree with your first assertion (although that depends on what we mean by "significant" here), and I certainly disagree with your second. We simply have not been able to detect or induce such an effect. Here, too, there are theoretical results showing that increased expressiveness cannot lead to cheaper reasoning, which is not an obvious result even in the worst case (because there are far fewer "compressible" programs than non compressible ones).
>We know the effect of different programming constructs on expressive compression and reasoning costs. The closer we get to the limit, the less benefit we can have.
>Here, too, there are theoretical results showing that increased expressiveness cannot lead to cheaper reasoning, which is not an obvious result even in the worst case (because there are far fewer "compressible" programs than non compressible ones).
I find that very interesting. Do you have a source for it?
>Here, too, there are theoretical results showing that increased expressiveness cannot lead to cheaper reasoning, which is not an obvious result even in the worst case (because there are far fewer "compressible" programs than non compressible ones).
I find that very interesting. Do you have a source for it?
Yes. Some interesting papers are:
* http://www.lsv.fr/Publis/PAPERS/PDF/Sch-aiml02.pdf
* http://www.lsv.fr/Publis/PAPERS/PDF/DLS-jcss-param.pdf
* https://www.cis.upenn.edu/~alur/Zohar03.pdf
What's important to put those results in context for those who are not familiar with the subject is that, in the context of complexity theory, the "model checking problem" does not refer to the complexity of a particular model checker algorithm, but to the inherent complexity of the problem of deciding whether a program M satisfies some property 𝜑 (i.e. whether M is a model of 𝜑 in the formal logic sense). The "model checking problem" is the closest to a mathematical description of what we usually mean when we talk of "reasoning about a program."
I've summarized these results and more here: https://pron.github.io/posts/correctness-and-complexity
* http://www.lsv.fr/Publis/PAPERS/PDF/Sch-aiml02.pdf
* http://www.lsv.fr/Publis/PAPERS/PDF/DLS-jcss-param.pdf
* https://www.cis.upenn.edu/~alur/Zohar03.pdf
What's important to put those results in context for those who are not familiar with the subject is that, in the context of complexity theory, the "model checking problem" does not refer to the complexity of a particular model checker algorithm, but to the inherent complexity of the problem of deciding whether a program M satisfies some property 𝜑 (i.e. whether M is a model of 𝜑 in the formal logic sense). The "model checking problem" is the closest to a mathematical description of what we usually mean when we talk of "reasoning about a program."
I've summarized these results and more here: https://pron.github.io/posts/correctness-and-complexity
> Because no large effect of programming language choice has been observed
Do you mind sharing a source for this? It seems trivial to generate a programming language (Brainf*ck) that performs very poorly.
A more likely statement: all commonly used languages are more or less on par.
Do you mind sharing a source for this? It seems trivial to generate a programming language (Brainf*ck) that performs very poorly.
A more likely statement: all commonly used languages are more or less on par.
I meant the latter. I just can't repeat all caveats every time I bring this subject up :) https://news.ycombinator.com/item?id=20074684
> It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much.
There's no reality here, just myth. That switching to Java would drive quality up makes at least as much sense as it driving it down.
If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality. As lack of evidence in favor of a big effect is evidence of its lack, the most probable explanation is that no such effect exists (it's possible a small effect exists, but we don't know in which language's favor). And, indeed, while there is a theory explaining (and even predicting ahead of time) why there is no big effect to the choice of the programming language, not only is there no empirical evidence suggesting there is such an effect, there is no theory to predict it, either. That language has a big effect on quality is, at this point, no more than wishful thinking among a minority of developers who are big programming language fans. It's a bedtime story. (I am not saying that language didn't ever have a big effect or that it never will, just that both theory and observation show that there isn't one currently among "reasonable" languages in common use; also, I have used and I like both Java and Clojure)
And if you're only referring to the priority of making the project more accessible, then I don't understand your comment at all. There are many, many more experienced Java developers than experienced Clojure developers, and experienced developers are at least as likely not to bother learning a new language in order to contribute to a project (although perhaps for different reasons). Picking a popular language makes the project more accessible to experienced developers.
What has been shown to drive quality up or down is process. If you have a good process, then making the project more accessible can help, and if you don't then you're screwed anyway.
There's no reality here, just myth. That switching to Java would drive quality up makes at least as much sense as it driving it down.
If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality. As lack of evidence in favor of a big effect is evidence of its lack, the most probable explanation is that no such effect exists (it's possible a small effect exists, but we don't know in which language's favor). And, indeed, while there is a theory explaining (and even predicting ahead of time) why there is no big effect to the choice of the programming language, not only is there no empirical evidence suggesting there is such an effect, there is no theory to predict it, either. That language has a big effect on quality is, at this point, no more than wishful thinking among a minority of developers who are big programming language fans. It's a bedtime story. (I am not saying that language didn't ever have a big effect or that it never will, just that both theory and observation show that there isn't one currently among "reasonable" languages in common use; also, I have used and I like both Java and Clojure)
And if you're only referring to the priority of making the project more accessible, then I don't understand your comment at all. There are many, many more experienced Java developers than experienced Clojure developers, and experienced developers are at least as likely not to bother learning a new language in order to contribute to a project (although perhaps for different reasons). Picking a popular language makes the project more accessible to experienced developers.
What has been shown to drive quality up or down is process. If you have a good process, then making the project more accessible can help, and if you don't then you're screwed anyway.
Peter Norvig made a comment here that agrees with you about process and architecture over language (he was talking about why he started using python instead of lisp)
"In terms of programming-in-the-large, at Google and elsewhere, I think that language choice is not as important as all the other choices: if you have the right overall architecture, the right team of programmers, the right development process that allows for rapid development with continuous improvement, then many languages will work for you; if you don't have those things you're in trouble regardless of your language choice."
https://news.ycombinator.com/item?id=1803815
"In terms of programming-in-the-large, at Google and elsewhere, I think that language choice is not as important as all the other choices: if you have the right overall architecture, the right team of programmers, the right development process that allows for rapid development with continuous improvement, then many languages will work for you; if you don't have those things you're in trouble regardless of your language choice."
https://news.ycombinator.com/item?id=1803815
If we're going to talk about empirical effects, we first need to agree on what quality means. Parts of the thread, including your comment, lead me to believe we sometimes mean an abstract, desirable but ineffable property--but then we're not speaking empirically.
Do you mean "defect rate" or something else?
Do you mean "defect rate" or something else?
I mean any reasonable metric you pick. The studies that have tried and failed to find a big effect used defect rate as a metric, but studies are not the only way to detect big effects. Because there are strong selective pressures in industry that can detect big effects, we can say that no big effect on the bottom line has been found.
> then the reality is that no big effect has been found for the choice of programming language on software quality
Substantiate that. I say it's garbage.
<https://www.cse.iitk.ac.in/users/karkare/courses/2010/cs653/..., and this BTW is from 1994 so this evidence has been around for ages
comments of note:
(2) The Ada solution was written by a lead programmer at NSWC; in this sense it represents the “control” group. The developer initially reported a line count of only 249; this was the numberof lines of imperativestatements as reported by theSun Adacompiler, anddid notinclude declarations, all of which were essential for proper execution. The line count of 767 is based on the actual code in and does not include lines with only termination characters.
(4) TheC++solutionwaswrittenbyanONRprogrammanagerafterhavingfirstwrittentheAwk solution described in the next paragraph. In addition to these 1105 lines of code, the developer also wrote a 595-line “test harness.” No development times were reported [Note: C++ has evolved a lot since then, but perhaps so has haskell]
(10) Intermetrics, independently and without the knowledge of NSWC or Yale University, con-ductedan experimentof its own: theHaskell Report was given to anewly hiredcollege graduate, who was then given 8 days to learn Haskell. This new hire received no formal training, but was allowed to ask an experienced Haskell programmer questions as issues came up during the self study. After this training period, the new hire was handed the geo-server specification and asked to write aprototype in Haskell. Theresulting metrics shown in row 10 ofthe tableare perhaps the most stunning of the lot, suggesting the ease with which Haskell may be learned and effectively utilized.
Also mentioned was "The use of higher-order functions [in haskell] is noteworthy" so language features did help. Actual evidence.
> As lack of evidence in favor of a big effect is evidence of its lack
yeah right. Let me rephrase that for you - Lack of knowledge by an HN poster is not evidence of lack of evidence.
> What has been shown to drive quality up or down is process
Evidence please? My experience is that process can be rigorous but useless crap. My old boss said "process is not sufficient to produce quality, but it is a necessary"
There's plenty more in that PDF, please read it.
Substantiate that. I say it's garbage.
<https://www.cse.iitk.ac.in/users/karkare/courses/2010/cs653/..., and this BTW is from 1994 so this evidence has been around for ages
Language LOC Documentation Lines Dev time (hours)
(1)Haskell 85 465 10
(2)Ada 767 714 23
(3)Ada9X 800 200 28
(4)C++ 1105 130 –
(5)Awk/Nawk 250 150 –
(6)Rapide 157 0 54
(7)Griffin 251 0 34
(8)Proteus 293 79 26
(9)RelationalLisp 274 12 3
(10)Haskell 156 112 8
comments of note:
(2) The Ada solution was written by a lead programmer at NSWC; in this sense it represents the “control” group. The developer initially reported a line count of only 249; this was the numberof lines of imperativestatements as reported by theSun Adacompiler, anddid notinclude declarations, all of which were essential for proper execution. The line count of 767 is based on the actual code in and does not include lines with only termination characters.
(4) TheC++solutionwaswrittenbyanONRprogrammanagerafterhavingfirstwrittentheAwk solution described in the next paragraph. In addition to these 1105 lines of code, the developer also wrote a 595-line “test harness.” No development times were reported [Note: C++ has evolved a lot since then, but perhaps so has haskell]
(10) Intermetrics, independently and without the knowledge of NSWC or Yale University, con-ductedan experimentof its own: theHaskell Report was given to anewly hiredcollege graduate, who was then given 8 days to learn Haskell. This new hire received no formal training, but was allowed to ask an experienced Haskell programmer questions as issues came up during the self study. After this training period, the new hire was handed the geo-server specification and asked to write aprototype in Haskell. Theresulting metrics shown in row 10 ofthe tableare perhaps the most stunning of the lot, suggesting the ease with which Haskell may be learned and effectively utilized.
Also mentioned was "The use of higher-order functions [in haskell] is noteworthy" so language features did help. Actual evidence.
> As lack of evidence in favor of a big effect is evidence of its lack
yeah right. Let me rephrase that for you - Lack of knowledge by an HN poster is not evidence of lack of evidence.
> What has been shown to drive quality up or down is process
Evidence please? My experience is that process can be rigorous but useless crap. My old boss said "process is not sufficient to produce quality, but it is a necessary"
There's plenty more in that PDF, please read it.
> this evidence has been around for ages
Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a completely separate one.
> Substantiate that
I don't know how I can substantiate the lack of evidence, but here's a recent failed attempt to find evidence: https://arxiv.org/pdf/1901.10220.pdf
> Lack of knowledge by an HN poster is not evidence of lack of evidence.
True. While I have been following the subject for at least the past 20 years (and I have read your PDF several times already, including shortly after it was published), it is certainly possible I have missed something. If there's any evidence of a large effect you believe you've found, let me know.
> Evidence please?
Sure. Here's code review, for example (I don't have time to discuss each paper -- as they're quite different -- but they all paint a similar picture). BTW, while programming language studies are debating effects in the 0-15% range, these report effects in the 30-80%:
* What We Have Learned About Fighting Defects, 2002 -- https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf
* The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data, 2009 -- https://www.pitt.edu/~ckemerer/PSP_Data.pdf
* Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software, 2017 -- https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-...
* What Types of Defects Are Really Discovered in Code Reviews?, 2003 -- https://ieeexplore.ieee.org/document/4604671
* Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?, 2014 -- https://www.testroots.org/assets/papers/2014_beller_bacchell...
* The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects, 2014 -- https://dl.acm.org/citation.cfm?id=2597076
* Best Kept Secrets of Peer Code Review, 2013 -- https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets...
* Modern Code Review: A Case Study at Google, 2018 -- https://storage.googleapis.com/pub-tools-public-publication-...
If we summarize our current state of knowledge (not myth) it is this: process matters a lot; language matters little, if at all (with all the caveats I mentioned in other comments, such as diminishing returns etc.).
Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a completely separate one.
> Substantiate that
I don't know how I can substantiate the lack of evidence, but here's a recent failed attempt to find evidence: https://arxiv.org/pdf/1901.10220.pdf
> Lack of knowledge by an HN poster is not evidence of lack of evidence.
True. While I have been following the subject for at least the past 20 years (and I have read your PDF several times already, including shortly after it was published), it is certainly possible I have missed something. If there's any evidence of a large effect you believe you've found, let me know.
> Evidence please?
Sure. Here's code review, for example (I don't have time to discuss each paper -- as they're quite different -- but they all paint a similar picture). BTW, while programming language studies are debating effects in the 0-15% range, these report effects in the 30-80%:
* What We Have Learned About Fighting Defects, 2002 -- https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf
* The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data, 2009 -- https://www.pitt.edu/~ckemerer/PSP_Data.pdf
* Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software, 2017 -- https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-...
* What Types of Defects Are Really Discovered in Code Reviews?, 2003 -- https://ieeexplore.ieee.org/document/4604671
* Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?, 2014 -- https://www.testroots.org/assets/papers/2014_beller_bacchell...
* The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects, 2014 -- https://dl.acm.org/citation.cfm?id=2597076
* Best Kept Secrets of Peer Code Review, 2013 -- https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets...
* Modern Code Review: A Case Study at Google, 2018 -- https://storage.googleapis.com/pub-tools-public-publication-...
If we summarize our current state of knowledge (not myth) it is this: process matters a lot; language matters little, if at all (with all the caveats I mentioned in other comments, such as diminishing returns etc.).
You're moving goalposts. You denied language mattered:
> then the reality is that no big effect has been found for the choice of programming language on software quality
Then I show some evidence giving small programs rapidly developed, which you then say doesn't count because it's 'tiny' and 'prototyping'. You also don't define quality which gives you lots of wriggle room.
Nope, 1000 line programs aren't tiny. They aren't industry monsters but you can't dismiss them because it contradicts you. It's not proof, but it is strong evidence.
...arxiv paper... Interesting, thanks. It would have been helpful to have posted that in your original post.
However from <https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Mult..., and note this paper is not mentioned, as in not debunked, in the above arxiv paper.
"Bhattacharya et al. study four open-source projects which use C and C++, i.e., Firefox, Blender, VLC Media Player and MySQL to understand the impact of languages on software quality [20]. They compute several statistical measures while controlling for factors, such as developer competence and software process. They find that applications previously written in C are migrating to C++ and C++ code is often of higher quality, less prone to bugs, and easier to maintain than C code."
and from that same paper:
"As can be seen in Tables 8, 9, and 10, the mean de- fect density values for the C sets can be up to an order of magnitude higher than the mean values for the C++ sets."
> Sure. Here's code review...
I was talking about your claim about languages not mattering, I did not mention code review. All these papers are about code review. This is relevant to my code review comment (edit: I meant process comment), they don't say anything about language vs bugs (unless you wish to point out a paper that does).
> process matters a lot
And I agreed with you, I just said it didn't deliver quality, it just prepared the ground for it. With enough process you can close any hole in a language, but it gets exponentially expensive.
> then the reality is that no big effect has been found for the choice of programming language on software quality
Then I show some evidence giving small programs rapidly developed, which you then say doesn't count because it's 'tiny' and 'prototyping'. You also don't define quality which gives you lots of wriggle room.
Nope, 1000 line programs aren't tiny. They aren't industry monsters but you can't dismiss them because it contradicts you. It's not proof, but it is strong evidence.
...arxiv paper... Interesting, thanks. It would have been helpful to have posted that in your original post.
However from <https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Mult..., and note this paper is not mentioned, as in not debunked, in the above arxiv paper.
"Bhattacharya et al. study four open-source projects which use C and C++, i.e., Firefox, Blender, VLC Media Player and MySQL to understand the impact of languages on software quality [20]. They compute several statistical measures while controlling for factors, such as developer competence and software process. They find that applications previously written in C are migrating to C++ and C++ code is often of higher quality, less prone to bugs, and easier to maintain than C code."
and from that same paper:
"As can be seen in Tables 8, 9, and 10, the mean de- fect density values for the C sets can be up to an order of magnitude higher than the mean values for the C++ sets."
> Sure. Here's code review...
I was talking about your claim about languages not mattering, I did not mention code review. All these papers are about code review. This is relevant to my code review comment (edit: I meant process comment), they don't say anything about language vs bugs (unless you wish to point out a paper that does).
> process matters a lot
And I agreed with you, I just said it didn't deliver quality, it just prepared the ground for it. With enough process you can close any hole in a language, but it gets exponentially expensive.
> You're moving goalposts.
Only if you insist on uncharitable reading. I'm talking about software development; if I mistakenly assumed that could be left implicit, I'm sorry. And ~500 line programs are positively minuscule. JQuery is ~50KLOC, and an average business system is ~5MLOC. We're talking four orders of magnitude between those prototypes and a rather average industry system size (systems commonly run to tens and even more than 100 MLOC).
> It's not proof, but it is strong evidence.
I don't think it matters if it's strong or weak evidence, as it's not even about software development.
> the C sets can be up to an order of magnitude higher than the mean values for the C++ sets.
As I mentioned in another comment, C is not a good example. It's a ~50-year-old language, and the theory that correctly predicted that languages won't make a big difference was based on diminishing returns. I.e. not that no two languages have ever been or could ever have a big difference, but that over time the ability to affect quality with language would severely diminish. The original prediction of that theory was that no 10x improvement would be made by a single language improvement over a decade, and was called overly pessimistic by PL fans. It's not been over thirty years, and we have doubtfully made a 3x boost with all language features combined.
If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric.
> I just said it didn't deliver quality
But those papers show that, unlike language, process does have a big impact on quality.
> With enough process you can close any hole in a language, but it gets exponentially expensive.
You are now making an unsubstantiated claim that contradicts a substantiated claim. Programming languages (with the caveats above) have not been found to have a big effect (nor is there a theory that suggests they do), while process has.
Only if you insist on uncharitable reading. I'm talking about software development; if I mistakenly assumed that could be left implicit, I'm sorry. And ~500 line programs are positively minuscule. JQuery is ~50KLOC, and an average business system is ~5MLOC. We're talking four orders of magnitude between those prototypes and a rather average industry system size (systems commonly run to tens and even more than 100 MLOC).
> It's not proof, but it is strong evidence.
I don't think it matters if it's strong or weak evidence, as it's not even about software development.
> the C sets can be up to an order of magnitude higher than the mean values for the C++ sets.
As I mentioned in another comment, C is not a good example. It's a ~50-year-old language, and the theory that correctly predicted that languages won't make a big difference was based on diminishing returns. I.e. not that no two languages have ever been or could ever have a big difference, but that over time the ability to affect quality with language would severely diminish. The original prediction of that theory was that no 10x improvement would be made by a single language improvement over a decade, and was called overly pessimistic by PL fans. It's not been over thirty years, and we have doubtfully made a 3x boost with all language features combined.
If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric.
> I just said it didn't deliver quality
But those papers show that, unlike language, process does have a big impact on quality.
> With enough process you can close any hole in a language, but it gets exponentially expensive.
You are now making an unsubstantiated claim that contradicts a substantiated claim. Programming languages (with the caveats above) have not been found to have a big effect (nor is there a theory that suggests they do), while process has.
[deleted]
> I'm talking about software development
And the value different languages may add is entirely about software development.
> I don't think it matters if it's strong or weak evidence, as it's not even about software development.
Programming languages are not about software development? Oh do go on.
> As I mentioned in another comment, C is not a good example.
A study of C vs C++ strongly indicates your claim about languages is false and suddenly C is not a good example? I'm trying to believe I'm just misunderstanding you but it's getting more difficult.
> The original prediction of that theory was that no 10x improvement would be made by a single language improvement
This isn't what you said. Let me remind you "If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality"
The goalposts aren't being moved, you've just bought them plane tickets to barbados.
> If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric
Another claim. Show me the study that says that.
> But those papers show that, unlike language, process does have a big impact on quality.
It can if done properly; it does not do so automatically. You can have code reviews that are of little use because you have no spec to review the code against. I have been in that very position. I don't dispute the value of process but it's a necessary but not sufficient condition to bring about quality. I also agree code reviews are good, if done properly.
ME >> With enough process you can close any hole in a language, but it gets exponentially expensive.
YOU > You are now making an unsubstantiated claim that contradicts a substantiated claim.
If I use C I have to worry about garbage leaks, double-freeing pointers, out-of-bounds accesses etc that can be detected with code reviews. If I use python, I never have any of these problems so a code review need not check for these. That's a lot cheaper.
(Edit: weird stuff happening to my post, may appear twice)
And the value different languages may add is entirely about software development.
> I don't think it matters if it's strong or weak evidence, as it's not even about software development.
Programming languages are not about software development? Oh do go on.
> As I mentioned in another comment, C is not a good example.
A study of C vs C++ strongly indicates your claim about languages is false and suddenly C is not a good example? I'm trying to believe I'm just misunderstanding you but it's getting more difficult.
> The original prediction of that theory was that no 10x improvement would be made by a single language improvement
This isn't what you said. Let me remind you "If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality"
The goalposts aren't being moved, you've just bought them plane tickets to barbados.
> If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric
Another claim. Show me the study that says that.
> But those papers show that, unlike language, process does have a big impact on quality.
It can if done properly; it does not do so automatically. You can have code reviews that are of little use because you have no spec to review the code against. I have been in that very position. I don't dispute the value of process but it's a necessary but not sufficient condition to bring about quality. I also agree code reviews are good, if done properly.
ME >> With enough process you can close any hole in a language, but it gets exponentially expensive.
YOU > You are now making an unsubstantiated claim that contradicts a substantiated claim.
If I use C I have to worry about garbage leaks, double-freeing pointers, out-of-bounds accesses etc that can be detected with code reviews. If I use python, I never have any of these problems so a code review need not check for these. That's a lot cheaper.
(Edit: weird stuff happening to my post, may appear twice)
> This isn't what you said.
I can't repeat all the caveats every time I say something. I assume readers believe my comments are at least reasonable, even if they disagree with them. For convenience, I restated the current state of knowledge more precisely in my previous comment. If you're interested in what I have to say, assume I'm not an idiot, and perhaps we could have an interesting discussion, and if you think I'm an idiot, then there's no point in arguing at all.
> Show me the study that says that.
Show me the study that says otherwise. I don't need to provide evidence for the lack of an effect (although I did). The lack of an effect is, at the very least, the default hypothesis in this case (as in many others) as there is no theory suggesting we should believe otherwise (while a theory that explains why languages increasingly have smaller effects has made correct predictions).
> It can if done properly
That's your hypothesis. We've been trying to find evidence of that, or others like it, for a long time -- both in academia and in industry -- without success. We simply do not observe that the choice of language today (among reasonable ones etc.) has a big effect, either in studies or in industry practice. But if you've found a language that can drastically reduce software development costs and/or increase quality at scale, that discovery can be easily translated to billions of dollars. Go ahead and make them.
I can't repeat all the caveats every time I say something. I assume readers believe my comments are at least reasonable, even if they disagree with them. For convenience, I restated the current state of knowledge more precisely in my previous comment. If you're interested in what I have to say, assume I'm not an idiot, and perhaps we could have an interesting discussion, and if you think I'm an idiot, then there's no point in arguing at all.
> Show me the study that says that.
Show me the study that says otherwise. I don't need to provide evidence for the lack of an effect (although I did). The lack of an effect is, at the very least, the default hypothesis in this case (as in many others) as there is no theory suggesting we should believe otherwise (while a theory that explains why languages increasingly have smaller effects has made correct predictions).
> It can if done properly
That's your hypothesis. We've been trying to find evidence of that, or others like it, for a long time -- both in academia and in industry -- without success. We simply do not observe that the choice of language today (among reasonable ones etc.) has a big effect, either in studies or in industry practice. But if you've found a language that can drastically reduce software development costs and/or increase quality at scale, that discovery can be easily translated to billions of dollars. Go ahead and make them.
So, you’re saying the quality of the code dropped with the move to java?
Could there be other reasons? Maybe they rushed?
Could there be other reasons? Maybe they rushed?
He is not saying Java caused it, just that the priority was cheaper labor, and a more popular language makes that more likely.
This is in the context of software development in general, not this specific project per se.
This is in the context of software development in general, not this specific project per se.
As a Clojure enthusiast who has never been able to find a FT job using the language, I find it hard to believe that the demand for Clojure devs outstrips the supply.
And even if you have to hire devs who don't already know the language I really don't understand the difficulty in learning Lisp - few languages are simpler or easier to grok.
And even if you have to hire devs who don't already know the language I really don't understand the difficulty in learning Lisp - few languages are simpler or easier to grok.
I didn't have any problems finding work in Clojure when I was last looking (Aug-Sept 2018) with offers in four locales across the US. I was not able to find remote work at comparable salary for a US based org. (Some context) I have 7 years experience using Clojure professionally across 3 orgs (16 years overall) and was looking for individual contributor roles.
Some things I observed about the Clojure companies:
- Their HR seemed slower than the non-Clojure companies I was dealing with. I don't think this has anything to do with Clojure, but I am no longer working in Clojure full-time because the most credible offers took too long and I've found non-Clojure work I love.
- Outside of trading firms, I did the most whiteboard coding questions with Clojure firms compared to shops looking to hire for Python or Elixir. I like whiteboard work, so this was fine for me.
- 6 Clojure companies, 4 different conccurrent processing models and 6 different service frameworks. Neither of these is necessarily bad, but the community was much more fractured than I realized.
- Of the 6 Clojure orgs, 4 were actively hiring (had open roles) and 2 were passively hiring whenever someone materialized on IRC/Slack/Twitter. This leads me to think if you want a Clojure job, you should probably reach out to people you know inside Clojure orgs and, barring that, the HR people at these orgs directly. This isn't a Clojure-specific thing, but worth keeping in mind.
- 4 of 6 didn't seem to have dedicates DevOps, leading to them looking for candidates also familiar with deploying, monitoring, and debugging Linux hosts. I think I was given heavier weight at some orgs because of a background doing this type of work for Clojure stacks. Might be an angle worth looking into if you don't have production Clojure but you do have Linux and Java deployment/monitoring/debugging/tuning experience.
Some things I observed about the Clojure companies:
- Their HR seemed slower than the non-Clojure companies I was dealing with. I don't think this has anything to do with Clojure, but I am no longer working in Clojure full-time because the most credible offers took too long and I've found non-Clojure work I love.
- Outside of trading firms, I did the most whiteboard coding questions with Clojure firms compared to shops looking to hire for Python or Elixir. I like whiteboard work, so this was fine for me.
- 6 Clojure companies, 4 different conccurrent processing models and 6 different service frameworks. Neither of these is necessarily bad, but the community was much more fractured than I realized.
- Of the 6 Clojure orgs, 4 were actively hiring (had open roles) and 2 were passively hiring whenever someone materialized on IRC/Slack/Twitter. This leads me to think if you want a Clojure job, you should probably reach out to people you know inside Clojure orgs and, barring that, the HR people at these orgs directly. This isn't a Clojure-specific thing, but worth keeping in mind.
- 4 of 6 didn't seem to have dedicates DevOps, leading to them looking for candidates also familiar with deploying, monitoring, and debugging Linux hosts. I think I was given heavier weight at some orgs because of a background doing this type of work for Clojure stacks. Might be an angle worth looking into if you don't have production Clojure but you do have Linux and Java deployment/monitoring/debugging/tuning experience.
The demand and supply for Clojure both exist, but they are not equally distributed geographically.
Things are progressing, but the cultural addiction to shipping meat around is still too high in my opinion. Geography is a weird and often arbitrary constraint for knowledge work.
> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.
I think that is why Kotlin and Typescript are doing so well generally, they build on top of already established bases offering good new things without throwing out everything else.
The trade off is they are constrained from how far they can diverge from their roots but on the flip side that is a strength as well.
I like both, they feel productive without feeling totally alien.
I think that is why Kotlin and Typescript are doing so well generally, they build on top of already established bases offering good new things without throwing out everything else.
The trade off is they are constrained from how far they can diverge from their roots but on the flip side that is a strength as well.
I like both, they feel productive without feeling totally alien.
Same thing I repeat again and again: There is a false cost assigned to learning a language. Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. We've got to work at showing developers those benefits early; it's as important to creating software effectively as any other engineer's basic toolkit.
I suspect programming has something similar to Gear/Plugin Acquisition Syndrome in photography and music. It's easy to let tinkering with new toys stand in for doing the actual work of creating.
At some point, you have to sit down and do the work with the tools you have if you want to get anything done. If I were to get serious about programming, I would use C# or Go because I found them the most accessible after trying lots of languages. There's nothing wrong with finding a niche.
At some point, you have to sit down and do the work with the tools you have if you want to get anything done. If I were to get serious about programming, I would use C# or Go because I found them the most accessible after trying lots of languages. There's nothing wrong with finding a niche.
True, but I expect you to know which two or three /distinct/ tools you can work with effectively, and which situations they are most suitable for.
Not only that, but knowing multiple languages has a multiplicative effect, massively expanding how you think of programming in all the languages you know, even the stable standbys like C#.
Not only that, but knowing multiple languages has a multiplicative effect, massively expanding how you think of programming in all the languages you know, even the stable standbys like C#.
> Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think.
and yet... almost every time I've come in to a project where developers got past those boundaries.... they inevitably recreated patterns/idioms from tech 1 in to tech 2, and make tech 2 horrifically bad, and made it even harder for anyone who actually knew tech 2 to come in to the project and help.
adding other languages is more than just 'get over it' - without guidance from experienced people who can understand good/efficient ways to model the problem domains unique to a business idiomatically in the new tech, you're very likely adding even more technical debt that you won't even know about for months/years. And... IME, the "let's use new stack X" are moved on to another project or company by the time the technical debt is apparent to everyone.
and yet... almost every time I've come in to a project where developers got past those boundaries.... they inevitably recreated patterns/idioms from tech 1 in to tech 2, and make tech 2 horrifically bad, and made it even harder for anyone who actually knew tech 2 to come in to the project and help.
adding other languages is more than just 'get over it' - without guidance from experienced people who can understand good/efficient ways to model the problem domains unique to a business idiomatically in the new tech, you're very likely adding even more technical debt that you won't even know about for months/years. And... IME, the "let's use new stack X" are moved on to another project or company by the time the technical debt is apparent to everyone.
Most CS programs have a course on functional programming or programming paradigms including functional programming.
I don't understand why it would be a challenge to pick up Clojure.
I don't understand why it would be a challenge to pick up Clojure.
What learnt in school is insufficient to meet with all real world challenges. To claim as a developer, you need to understand the internals, that is the ability to poke and trace to the root of the problem deep underneath.
Debugging in FP feels different to most people, where our thoughts is tuned to the procedural model with observing states and form our hypothesis from there.
Another disadvantage with niche programming language is not enough Stack Overflow help there. That alone will drive most people away.
Debugging in FP feels different to most people, where our thoughts is tuned to the procedural model with observing states and form our hypothesis from there.
Another disadvantage with niche programming language is not enough Stack Overflow help there. That alone will drive most people away.
I don't know how it is elsewhere, but I saw many (most?) students struggle with those classes. These are people who have learned to successfully program, but they struggled quite significantly with functional languages (Haskell in this case). Even when they pass, many of them probably wouldn't want to pick up a functional language afterwards.
But Scheme and Common Lisp (and its dialects) are drastically easier to learn than Haskell.
Furthermore, I think it also has to do with the order in which these courses are taught. When I did a small introduction to FP at my university, a lot of people who had no prior experience in programming picked up Haskell much more easily than those who already had a background in imperative/OO programming.
(Though that strengthens the point - FP is often quite a leap for programmers who have used imperative languages for a long time.)
Furthermore, I think it also has to do with the order in which these courses are taught. When I did a small introduction to FP at my university, a lot of people who had no prior experience in programming picked up Haskell much more easily than those who already had a background in imperative/OO programming.
(Though that strengthens the point - FP is often quite a leap for programmers who have used imperative languages for a long time.)
People have limited time in their lives. If you want to recruit volunteers to your open source project it has to be in a language they either already know or are interested in learning. It may not be “hard” to learn Clojure, but the potential contributor may consider the time and mental energy spent fumbling around frustrated with all the off-putting sigils and different brackets to be better used learning a language with brighter prospects.
Many cs courses teach ML style functional programming, not Lisp or Clojure
FWIW, I have a very different read on this, based on both my experience with Clojure and some of the leading Storm-using orgs (and the Apache process).
Firstly: Clojure isn't actually a problem. We hired people who are highly junior (one who even refused to call themselves a developer) and taught them Clojure. It was fine. It is true that you need a strong dev lead to guide them through that, but honestly, if you don't have a strong dev lead your Python/Java project is screwed too.
Secondly: look at Storm's increased success at Java data processing sweatshop^H^H^H^H^H^Hconsultancies. Of course "can write Java" is what you're going to get if most of your business revolves around the lowest bidder.
Firstly: Clojure isn't actually a problem. We hired people who are highly junior (one who even refused to call themselves a developer) and taught them Clojure. It was fine. It is true that you need a strong dev lead to guide them through that, but honestly, if you don't have a strong dev lead your Python/Java project is screwed too.
Secondly: look at Storm's increased success at Java data processing sweatshop^H^H^H^H^H^Hconsultancies. Of course "can write Java" is what you're going to get if most of your business revolves around the lowest bidder.
A more glass-half-full view of it is that it's a magnificent success for Clojure.
Yes, it got redone in Java, but the fact is that Clojure took it from nothing to one of the world's premier frameworks for distributed computing - something that might have been impossible with the same resources if Java was its starting language. And because of the commonality of the underlying JVM, choosing Clojure allowed the functional approach with far less risk because the transition later was (I am assuming) much less risky and smoother.
Looking at it in that light, this would be a huge boon to people considering starting new projects in Clojure (or possibly other JVM languages): it proves if they succeed and get so popular that moving to Java is the right thing, it can be done.
Yes, it got redone in Java, but the fact is that Clojure took it from nothing to one of the world's premier frameworks for distributed computing - something that might have been impossible with the same resources if Java was its starting language. And because of the commonality of the underlying JVM, choosing Clojure allowed the functional approach with far less risk because the transition later was (I am assuming) much less risky and smoother.
Looking at it in that light, this would be a huge boon to people considering starting new projects in Clojure (or possibly other JVM languages): it proves if they succeed and get so popular that moving to Java is the right thing, it can be done.
How long does it take a strong developer to learn Clojure? Is it really a big a barrier to new contributors?
Presumably people who consider contributing to the open source have some spare time and love programming. Wouldn't they find it exciting to contribute while also learning a cool new language?
Presumably people who consider contributing to the open source have some spare time and love programming. Wouldn't they find it exciting to contribute while also learning a cool new language?
My experience from hiring and training co-op students for many years is that it takes around 2 weeks to learn with a bit of guidance. when a co-op student with little to know programming experience can learn Clojure in that time I find it surreal that there are professional developers who cannot.
Maybe, but learning a new language by contributing to a large project built in that language is asking for trouble. Programming languages aren’t just syntax and stdlib. The relevant idioms and patterns may be significantly different. Even making small changes can be hard and intimidating.
And then, even if you wanted to learn something new, to a lot of people, it matters what that something is. Unless the person is really set on contributing to Storm specifically, then the barrier of Clojure is still significant. They may prefer to spend their new language learning bandwidth on something with more momentum.
And then, even if you wanted to learn something new, to a lot of people, it matters what that something is. Unless the person is really set on contributing to Storm specifically, then the barrier of Clojure is still significant. They may prefer to spend their new language learning bandwidth on something with more momentum.
another interesting part:
The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
Sometimes that backfires.
- Clojure dev
- Clojure dev
> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.
Describes my dating life pretty well.
Describes my dating life pretty well.
To me, this makes total sense as the project moved to Apache. Obviously, much more people will be able to consider contributing when it's in Java. Apache goal is sustainability and long-term viability, and Java would work better for that.
I also consider this a success story for Clojure. It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years and benefit thousands of developers until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.
I also consider this a success story for Clojure. It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years and benefit thousands of developers until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.
(I think the HN title was edited since your comment; I think an earlier title highlighted the move from Clojure to Java.)
> It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years [...] until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.
That general story, first heard from Paul Graham, about projects starting with Lisp and being successful that way, before eventually being rewritten, works for me.
I've been hoping at least a couple college startups would be inspired by this to use Racket initially, but if they have, I haven't heard of it. (I speculate that the current FAANG feeder emphasis among CS students hasn't helped. Who has time to play with a Lisp, when the now-all-important whiteboard interviews won't use it.)
> Obviously, much more people will be able to consider contributing when it's in Java.
If you have a company, and you want to hire Lisp people (whether it's Clojure, CL, or Scheme/Racket), I think you can probably hire people, because Lisp people like getting paid to use Lisp.
If you're looking for unpaid/volunteer contributions to an open source project, there's all sorts of things that affect that, and it's not unusual to get little-to-no contributions.
> It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years [...] until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.
That general story, first heard from Paul Graham, about projects starting with Lisp and being successful that way, before eventually being rewritten, works for me.
I've been hoping at least a couple college startups would be inspired by this to use Racket initially, but if they have, I haven't heard of it. (I speculate that the current FAANG feeder emphasis among CS students hasn't helped. Who has time to play with a Lisp, when the now-all-important whiteboard interviews won't use it.)
> Obviously, much more people will be able to consider contributing when it's in Java.
If you have a company, and you want to hire Lisp people (whether it's Clojure, CL, or Scheme/Racket), I think you can probably hire people, because Lisp people like getting paid to use Lisp.
If you're looking for unpaid/volunteer contributions to an open source project, there's all sorts of things that affect that, and it's not unusual to get little-to-no contributions.
> I think the HN title was edited since your comment; I think an earlier title highlighted the move from Clojure to Java.
This is correct, it was “Storm ditches Clojure in favor for Java”.
This is correct, it was “Storm ditches Clojure in favor for Java”.
"Storm ditches Clojure in favor of Java in its core". A moderator changed it, in accordance with the site guideline: "Please use the original title, unless it is misleading or linkbait; don't editorialize."
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
> I also consider this a success story for Clojure.
It is a success story for Clojure, but this move is a big negative feedback for the language. A team starting out on an open-source project will be mighty reluctant to start it with Clojure; because it might get rewritten not so much into the future. That is not good news for Clojure
It is a success story for Clojure, but this move is a big negative feedback for the language. A team starting out on an open-source project will be mighty reluctant to start it with Clojure; because it might get rewritten not so much into the future. That is not good news for Clojure
If, five years (or what-have-you) down the line—when you've pretty much enumerated all use-cases for the project, solidified what it's supposed to do and how, learned from your mistakes, and so on—you can't come up with a better design, something is off. With all that knowledge, there should be plenty of space for performance optimizations and pruning of vestigial stuff for a non-trivial project.
As for switching to Java, that makes perfect sense to me if that is what the contributors prefer. If you're going to make a big rewrite, you might as well change the language to one you prefer when you have the opportunity.
As someone who spends 100% of dev time in Clojure, I'm quite happy to see this development. Dipping into Java isn't uncommon for optimizations in Clojure, so this is like someone taking the time to do a massive under-the-hood optimization from the point of view of a Clojure user. I doubt it changes the project's status much in the Clojure community. Perhaps it'll even see an uptake in use.
What it doesn't say much about is Java vs. Clojure in my opinion. Different contributors, with different amounts of pertinent knowledge, and years apart. I'm not sure how you would control for those variables in a comparison. It should be read more as, "we put in a heck of an effort to make this thing faster and more useful", and kudos for that.
As for switching to Java, that makes perfect sense to me if that is what the contributors prefer. If you're going to make a big rewrite, you might as well change the language to one you prefer when you have the opportunity.
As someone who spends 100% of dev time in Clojure, I'm quite happy to see this development. Dipping into Java isn't uncommon for optimizations in Clojure, so this is like someone taking the time to do a massive under-the-hood optimization from the point of view of a Clojure user. I doubt it changes the project's status much in the Clojure community. Perhaps it'll even see an uptake in use.
What it doesn't say much about is Java vs. Clojure in my opinion. Different contributors, with different amounts of pertinent knowledge, and years apart. I'm not sure how you would control for those variables in a comparison. It should be read more as, "we put in a heck of an effort to make this thing faster and more useful", and kudos for that.
Pegging Clojure as a prototyping language relative to Java is hardly encouraging.
Oh no, I'm saying it's another usecase, not the only one. And yet, it is a pattern I follow quite often in my mixed Clojure/Java projects: Clojure code is much easier to tinker with, take to new directions, rapidly modify and see the results. But once I settle with an approach/algorithm, I rewrite some things to Java for performance, or for easier interop with other Java/Scala teams, or because the imperative mutable code looks much better in Java.
So, you can say that Clojure among everything else is my IDE to write Java code. Is that a bad suit for a language? Depends on your perspective, but I personally am very happy that I have Clojure by my side.
So, you can say that Clojure among everything else is my IDE to write Java code. Is that a bad suit for a language? Depends on your perspective, but I personally am very happy that I have Clojure by my side.
I think it’s noteworthy that whatever the benefits of clojure it didn’t outweigh the adoption issue.
It’s quite a mature language so I’m not sure that bodes we’ll for its prospects this late in the game.
It’s quite a mature language so I’m not sure that bodes we’ll for its prospects this late in the game.
Whatever the benefits of Clojure are, they were enough for Nathan Marz and first contributors to write Storm in it and succeed. Note that it's not an argument about impossibility of writing something like that in Java (see Spark, Flink, Heron, etc). But at that exact moment in time, a Clojure Storm was created and became useful to many people for years. There were many more Java programmers than Clojure programmers then and now, yet the Java coomunity didn't produce their own Storm first.
I think it speaks of something.
I think it speaks of something.
If you can have the foresight to know how your project will be used many years down the line, sure, make technical decisions based on that :-)
(Personally, I think the change is rather evidence of the primary drivers behind Storm development in 2019, and... that's not a good thing.)
(Personally, I think the change is rather evidence of the primary drivers behind Storm development in 2019, and... that's not a good thing.)
A Java shop inherited a Clojure project and rewrote it in their preferred language. It's silly to read more than that into this.
I’d agree if it didn’t keep happening .
It doesn't. There are plenty of Clojure projects out there and nobody's rewriting them in other languages. There are also plenty of companies nowadays that have been using Clojure for years. The feedback from them is overwhelmingly positive.
Rewrites between languages keep happening in all kinds of permutations. I think pessimism or optimism from just headlines and announcements is more about what you read into it.
It reminds me of the “why we switched from Mongo to X” or “why we switched from Ruby to Y” trends some years ago.
I saw Clojure become popular long before it was ready for the limelight: it definitely wasn’t pragmatism driving it. It was a fetish for syntax and paradigm.
I saw Clojure become popular long before it was ready for the limelight: it definitely wasn’t pragmatism driving it. It was a fetish for syntax and paradigm.
What if Clojure is a language that bodes well especially early in a project’s lifecycle?
I think it’s very difficult to motivate going out of your way to hire decent devs that know clojure and java only to throw away the clojure some time down the road.
Begs the question, why not just write it in Java? Then at least its more likely to be a refactor down the road rather than a rewrite.
I think clojure ought to be a production ready language that scales well. That’s what it was designed to be. However lisp seems to dichotomise devs into those that get it and those that don’t and thus alienates many would be team members.
Begs the question, why not just write it in Java? Then at least its more likely to be a refactor down the road rather than a rewrite.
I think clojure ought to be a production ready language that scales well. That’s what it was designed to be. However lisp seems to dichotomise devs into those that get it and those that don’t and thus alienates many would be team members.
See my response inspired by your comment and agumonkey here https://news.ycombinator.com/item?id=20074532
I am drawn to the idea of scalable languages or languages with zero friction for interop.
This thread on Pallene is interesting https://news.ycombinator.com/item?id=18038619
I am drawn to the idea of scalable languages or languages with zero friction for interop.
This thread on Pallene is interesting https://news.ycombinator.com/item?id=18038619
It is interesting to read the history of Storm from the original author (Nathan Marz) himself: http://nathanmarz.com/blog/history-of-apache-storm-and-lesso...
Clojure allowed him to build the initial Storm release by himself in only 5 months.
> I made all of Storm's APIs in Java, but implemented Storm in Clojure. By keeping Storm's APIs 100% Java, Storm was ensured to have a very large amount of potential users. By doing the implementation in Clojure, I was able to be a lot more productive and get the project working sooner.
Also interesting is that Nathan has now founded a startup working on something which sounds like the next evolution of Storm, and he's building it in Clojure again:
https://redplanetlabs.com/
They recently got 5M in funding and are hiring. Building a Clojure team for it.
Personally, I feel the Storm 2.0 release rewrite to Java is really just a case of the new maintainers not knowing Clojure very well, and being primarily Java devs, looking for contributions from people using Storm, which tend itself to be mostly Java shops. Clojure is not quick to pick up, and very different from Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and no Clojure or Lisp experience won't be able to contribute as easily. And that's fine, and possibly best for Storm's future now that Nathan has moved on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't at all surprising.
Clojure allowed him to build the initial Storm release by himself in only 5 months.
> I made all of Storm's APIs in Java, but implemented Storm in Clojure. By keeping Storm's APIs 100% Java, Storm was ensured to have a very large amount of potential users. By doing the implementation in Clojure, I was able to be a lot more productive and get the project working sooner.
Also interesting is that Nathan has now founded a startup working on something which sounds like the next evolution of Storm, and he's building it in Clojure again:
https://redplanetlabs.com/
They recently got 5M in funding and are hiring. Building a Clojure team for it.
Personally, I feel the Storm 2.0 release rewrite to Java is really just a case of the new maintainers not knowing Clojure very well, and being primarily Java devs, looking for contributions from people using Storm, which tend itself to be mostly Java shops. Clojure is not quick to pick up, and very different from Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and no Clojure or Lisp experience won't be able to contribute as easily. And that's fine, and possibly best for Storm's future now that Nathan has moved on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't at all surprising.
I struggle to see how you pick Java over Kotlin on the JVM these days. It's everything that Java is, minus everything Java shouldn't be, plus everything that Java should be.
Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it.
Kotlin's null safety alone is enough to prove you wrong here.
I find the whole discussion about null safety completely negligible.
Since when are nulls the biggest problem for developers? Null pointer exceptions are by far some the simplest problems to avoid and/or solve.
There are more serious arguments that can be made in favor of Kotlin.
Since when are nulls the biggest problem for developers? Null pointer exceptions are by far some the simplest problems to avoid and/or solve.
There are more serious arguments that can be made in favor of Kotlin.
Here here. I’ve heard “but null pointer errors!” countless times to justify one’s coding approaches (eg, strong typing, Maybe/Optional invasion, this or that PL, ...)
People will murder all kinds of productivity to avoid potential NPEs, when as you say they are generally the easiest bugs in the world to contend with.
It is beyond baffling to me, and given how often I hear it, it makes me question the dev world’s sanity.
People will murder all kinds of productivity to avoid potential NPEs, when as you say they are generally the easiest bugs in the world to contend with.
It is beyond baffling to me, and given how often I hear it, it makes me question the dev world’s sanity.
Lately I’ve seen code bases where the Java devs don’t return empty arrays or collections but return null instead. Of course they have to test for null in their calling code. I suggested to return empty collections and objects implementing the NullObject pattern. I think this will make the code more readable, simpler and more secure, but I was answered with skepticism.
Interestingly I agree w/ a preference for empty semantics over null semantics, but NOT for the reason "avoid NPE". Rather, I think minimizing branching/forking in the code logic is how complexity is reduced.
For this same reason I don't care for NullObject/Maybe/Optional type approaches which simply do not change the branching forking in your code:
For this same reason I don't care for NullObject/Maybe/Optional type approaches which simply do not change the branching forking in your code:
if (val == null) { ...logic... }
((Optional) val).ifNull(v -> ...logic...)
I'm dealing with an old code base in which methods tend to process exceptions by catching, logging and returning a null. The dev that wrote this code treated exceptions more like a rare-occurring annoyance rather than a signal for error, thus none of the calling methods had null check branching implemented. The amount of NPEs I had to fix in the past year is astounding.
Optional to me is a large leap forward. That being said, I also don't like its verbosity and wish for it to be improved somehow.
Optional to me is a large leap forward. That being said, I also don't like its verbosity and wish for it to be improved somehow.
Advocating programming constructs as a kind of ‘policing’ against bad developers doesn’t work for me.
It usually ends up at odds with better objectives & heavily taxes conscientious developers, for whom we should really be optimizing things imo.
It usually ends up at odds with better objectives & heavily taxes conscientious developers, for whom we should really be optimizing things imo.
Where was it suggested that nulls are the biggest problem for developers?
But they are a problem that happens, and in Kotlin, the compiler checks your nullable logic for you; instead of your users finding the errors at runtime.
But they are a problem that happens, and in Kotlin, the compiler checks your nullable logic for you; instead of your users finding the errors at runtime.
It was suggested that that feature is relevant enough to justify using kotlin over Java. However, I'd (and others in this very thread also have) argue that only slightly tips the scales towards Kotlin, while not solving the problem they were actually trying to solve here: reduce the barrier for contributing.
It just a way to conceal bad coding practices, which doesn’t prevent from having the same problem in a different form, just like GC in JVM cannot protect from all memory leaks. I can hardly remember a case when NPE was reported by some user in my projects.
You can easily get it on Java with PMD and other static checkers.
If you care about null safety, why not just use Scala? I've programmed Scala professionally for a long time and have never ever experienced a null pointer exception.
Correct null handling genuinely helps you write better, more correct, less bugged code.
It's not just syntax sugar.
It's not just syntax sugar.
Instal PMD and you are set.
[deleted]
The only Kotlin feature that it's really hard to live without is properties. Otherwise sure, it's better than Java 11, but I can live without all those niceties. And (non)nullable types even get in my way!
I just don't understand where there's syntax sugar for propeties in Java. There are lot of unneeded stuff added, that I don't really care about. But my classes are still full of autogenerated getters/setters. I recently tried to find some JEP and I did not find one. There was some talk around Java 7, but nothing since that. Nobody even wants to add properties support for Java?
I just don't understand where there's syntax sugar for propeties in Java. There are lot of unneeded stuff added, that I don't really care about. But my classes are still full of autogenerated getters/setters. I recently tried to find some JEP and I did not find one. There was some talk around Java 7, but nothing since that. Nobody even wants to add properties support for Java?
You kind of answered your own question: auto-generated getters/setters. Having a tolerable Java experience is so dependent on huge IDEs and other huge frameworks that people overlook language issues.
And (non)nullable types even get in my way!
Interesting how that goes...: the sibling comment to yours said the exact opposite. How does this feature get in your way?
Interesting how that goes...: the sibling comment to yours said the exact opposite. How does this feature get in your way?
1. Language problem:
2. Interoperation problem. A lot of Java code does not use @Nullable annotations, so all types are just that: platform types. So Kotlin nullability adds nothing to that.
if (this.prop != null) {
this.prop.something() // does not work
}
this.prop?.apply { prop ->
prop.something()
}
2. Interoperation problem. A lot of Java code does not use @Nullable annotations, so all types are just that: platform types. So Kotlin nullability adds nothing to that.
There’s only one case where properties may be useful - GUI forms mapped to domain-like model. In practically any other case it’s violation of one or more OO design principles. In Java you can live without them with a life full of joy.
Probably because in general they are a code smell. They are anti-OO.
That's just not true. If I'm writing simplest REST service, I have to return JSON responses. To return JSON response, I have to write class corresponding to that response. And that will be data class. If I'm interacting with database, I'm writing data mapping classes. And that will be data class. Those are normal code that every developer writes every day. If you think that it's anti-OO, then I don't care about that OO. I care about writing performant, concise and extensible software.
That statement is valid in both contexts you mention. You do not need properties for data mapping and serialization - it’s possible to write a good OO code without getters and setters.
If a useful feature is incompatible with OO, it's a problem with OO, not with the feature. Properties allow one to work with plain old data (not everything is meaningfully an object with behavior) without freezing the underlying implementation forever. Fewer (or maybe more decentralized) religious arguments like this is one of the few things I like about my job in C++. (Shame C++ doesn't have properties, these would come in handy in refactoring config handling code that I currently work on, but on the other hand the last thing C++ needs is yet another feature hidden under common syntax that may blow up allocations and runtime unexpectedly).
EDIT: to clarify: for example, I consider std::optional as an example of a very "un-OO" type that's immensely useful. In general I like the idea from functional programming that complex types can be treated no different than primitives, rather than the OO way of wrapping and uplifting everything until even integers have methods.
EDIT: to clarify: for example, I consider std::optional as an example of a very "un-OO" type that's immensely useful. In general I like the idea from functional programming that complex types can be treated no different than primitives, rather than the OO way of wrapping and uplifting everything until even integers have methods.
They want to use a language that has a large pool of interchangeable "x years of experience in language y" labour available.
As someone that uses both languages, I can't help but think that the OpenJDK will (hopefully) start implementing much of the Kotlin paradigms. Perhaps at some point Java & Kotlin would not differ too much. Let's see what the 6-month release cycle brings. That remains to be seen.
I guess you mean Java here, and yes some of them have a JEP defined.
Except there is no need to do. They already biased for maturity and long term maintainability. I don't think Kotlin is bad per say, but it feels untterly unnecessary.
Because until JVM turns into KVM, Kotlin is just another guest language on the JVM, without much to add as a company pushing their agenda and trying to use the JVM as a lever.
KVM might well turn out to be ART, but that is the extent of it.
KVM might well turn out to be ART, but that is the extent of it.
As someone who just started using Kotlin again, I'm curious what you mean by "agenda". A typical company trying to make products for money, or something else?
Yes, Kotlin as gateway drug into JetBrains products.
For example, the graphical debugger for Kotlin/Native is only available in Clion.
There is even a statement in that regard.
For example, the graphical debugger for Kotlin/Native is only available in Clion.
There is even a statement in that regard.
"While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute." - interesting phrasing. Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure. And those who know Java find Clojure difficult to learn? Clojure, in fact is quite easy to grasp - especially for someone coming from Java, as you would still be working on JVM and always have Java Interop. I guess same thing goes to someone coming from Erlang to Elixir, where you are still working in the familiar ecosystem.
> Clojure, in fact is quite easy to grasp - especially for someone coming from Java
I disagree on that point. In my experience as a programming language teacher, people often struggle to learn functional programming and even more to learn a Lisp, especially if they are used to a C-like language. And learning both at the same time is even more difficult. I liked working with Clojure but I have yet to meet someone who easily grasps the language and - more important - development patterns.
I disagree on that point. In my experience as a programming language teacher, people often struggle to learn functional programming and even more to learn a Lisp, especially if they are used to a C-like language. And learning both at the same time is even more difficult. I liked working with Clojure but I have yet to meet someone who easily grasps the language and - more important - development patterns.
Hello, this is an interesting take. Clojure has been a hobby of mine for some time but now I am a professional Clojure developper (clj/cljs). I knew the language before starting my new job but I found it really different than know the "platform": how a project is layed out, tests are written, some libs, how to pinpoint potential technical debt/suboptimal code accurately and so on... But I guess this would have been true if I had transition to any other new language.
Would you have any example of what your trainees stumbled upon or found difficult in the language? That would be useful I guess when we try to onboard new people on the project.
Have a nice day :)
Would you have any example of what your trainees stumbled upon or found difficult in the language? That would be useful I guess when we try to onboard new people on the project.
Have a nice day :)
> even more to learn a Lisp, especially if they are used to a C-like language
Ah, you're mixing up Clojure and Lisp. The restrictions in Clojure have nothing to do with Lisp tradition.
Someone used to a Lisp-like language will also struggle with Clojure.
Traditional Lisps like Common Lisp are multi-paradigm languages in which variables and most aggregate types are mutable.
They are not hard to pick up for someone with a background in C.
C has pass-by-value functions; so does proper, traditional Lisp.
C has mutable variables, arrays and structures; so does Lisp. Lisp data structures like lists or hash tables are easily understandable by a C programmer with a good background in data structures.
cons is like malloc-ing and filling-in a two-field struct, except that freeing is taken care of.
People have written Lisp systems in C (and written more than one book on that exact subject: Nils M. Holm just put a new one out: http://www.t3x.org/lsi/index.html), with a lot of the internals being written in C over the Lisp library functions. E.g. visiting and printing the elements of a list can literally look like this, in the internals of a C-based Lisp implementation:
Ah, you're mixing up Clojure and Lisp. The restrictions in Clojure have nothing to do with Lisp tradition.
Someone used to a Lisp-like language will also struggle with Clojure.
Traditional Lisps like Common Lisp are multi-paradigm languages in which variables and most aggregate types are mutable.
They are not hard to pick up for someone with a background in C.
C has pass-by-value functions; so does proper, traditional Lisp.
C has mutable variables, arrays and structures; so does Lisp. Lisp data structures like lists or hash tables are easily understandable by a C programmer with a good background in data structures.
cons is like malloc-ing and filling-in a two-field struct, except that freeing is taken care of.
People have written Lisp systems in C (and written more than one book on that exact subject: Nils M. Holm just put a new one out: http://www.t3x.org/lsi/index.html), with a lot of the internals being written in C over the Lisp library functions. E.g. visiting and printing the elements of a list can literally look like this, in the internals of a C-based Lisp implementation:
for (val iter = list; list; list = cdr(list))
print(car(list));
Hey, once you get over accepting a = a+1; without gagging it’s all downhill from there ;)
> Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure.
No, many more people know Java than Clojure, that's it. Whether it's the right decision for the future of the project, time will show, but I don't think it was taken lightly.
No, many more people know Java than Clojure, that's it. Whether it's the right decision for the future of the project, time will show, but I don't think it was taken lightly.
Also, java 8 and above is getting a lot leaner to use, api, linguistics .. so the gap with clojure is a bit less than before.
It will take years if not decades for these things to seep into regular programming practice in the Java world.
I still see people writing
I still see people writing
for(i = 0; i < something; i++) {
}
Really ? surprising.. (genuinely)
Most new contributors already know Java.
That's like saying going from any language to any language on the Unix platform is easy because the POSIX system calls are the same. In any half-decent language on Unix you can get access to a file descriptor and call fcntl() on it, so it's all the same.
It's not about the language, it's about the code... understanding Clojure code that someone else wrote is harder than writing your own.
I don't find that to be the case at all. I worked with Java for around a decade and I never contributed to a single open source project. Every time I'd see a bug or a feature I'd like to add, I'd open the source and it would be a nightmarish maze of classes and interfaces with business logic being sprinkled across them.
Meanwhile, I've successfully contributed to plenty of Clojure libraries I use, and many people contribute to my projects. Clojure code is far more direct and concise making it much easier to understand the intent. And majority of the code is written using pure functions that can be reasoned about in isolation. On top of that you have an editor integrated REPL, so you can just run any code you're not sure about as you're reading it.
Meanwhile, I've successfully contributed to plenty of Clojure libraries I use, and many people contribute to my projects. Clojure code is far more direct and concise making it much easier to understand the intent. And majority of the code is written using pure functions that can be reasoned about in isolation. On top of that you have an editor integrated REPL, so you can just run any code you're not sure about as you're reading it.
Apart from the language part , i am super excited about trying storm-2.0. In many ways the buzz in the ecosystem has been that storm was dead!. We run lot of prod workloads on storm and spark , storm has worked like a wonderful workhorse without quirks. The ecosystem has moved on , but i think storm may still continue having loyalists!
> The ecosystem has moved on
Just ouf of curiosity, what tools are people using instead of Storm?
Just ouf of curiosity, what tools are people using instead of Storm?
Flink is a strong competitor. Spark itself also claims to be a runner up with minibatching.
But streaming model is more widely adopted outside of frameworks. AWS Lambda or Fass platform can be viewed as streaming model with simple topology.
But streaming model is more widely adopted outside of frameworks. AWS Lambda or Fass platform can be viewed as streaming model with simple topology.
Flink is popular. I believe Samza also fills a similar niche.
There's also Kafka Streams which is suitable for many cases that do not warrant a full cluster with all the management and trouble required.
There's also Kafka Streams which is suitable for many cases that do not warrant a full cluster with all the management and trouble required.
I thought Heron was the Storm follow-up. I'm surprised to see a Storm 2.0.
Clojure just seems "too weird" to too many people, albeit for all the wrong reasons. Its syntax looks like a 6ft hurdle to newcomers, especially if they have lots of experience in C-synthax-languages.
But anyone who just /tries/ to jump it finds out the hurdle really isn't that high at all.
This dynamic will probably never change. Clojure will remain an acquired taste, which is fine, I guess. Those 'in the know' will happily continue to develop effectively with it.
This dynamic will probably never change. Clojure will remain an acquired taste, which is fine, I guess. Those 'in the know' will happily continue to develop effectively with it.
In this thread:
* Non-Java JVM languages are too different for people to use.
Also:
* It's not worth switching from Java to other languages because they're just Java with syntactic sugar.
* Non-Java JVM languages are too different for people to use.
Also:
* It's not worth switching from Java to other languages because they're just Java with syntactic sugar.
Clojure and Kotlin are two very different languages. I don’t see anyone making the broad statements you are calling out.
No, of course not.
I misread
> Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it.
and
> Also, java 8 and above is getting a lot leaner to use, api, linguistics .. so the gap with clojure is a bit less than before.
I misread
> Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it.
and
> Also, java 8 and above is getting a lot leaner to use, api, linguistics .. so the gap with clojure is a bit less than before.
Apache products have always been a result of a blind majority rule, and thanks to that many of them are a corporate-style incomprehensible mess. They favor the process instead of a result: Apache things need to show constant progress to be afloat.
Indeed, adopting Clojure for the developer product of this style is a dead end. Clojure is a minority rule. A slow tinkerer's and a lazy (in a good sense) practitioner's tool, whose aim is to survive in a real world, not to show a steady movement no matter what.
Indeed, adopting Clojure for the developer product of this style is a dead end. Clojure is a minority rule. A slow tinkerer's and a lazy (in a good sense) practitioner's tool, whose aim is to survive in a real world, not to show a steady movement no matter what.
These Talebian references got me thinking. Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java?
> Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java?
Since when trying to do better than using a tool designed for mass-producing software with dumb and cheap labor is "falling in love with ideals and tools"? It's kind of like saying that using an excavator over a bunch of people with shovels is "falling in love with ideals and tools". No, it just allows a single person to get more work done faster and better.
Our industry is weird. When recruiting, companies claim they want "the best of the best" and will often test you on ridiculous stuff. But when it comes to actual work, on industry scale, learning, growing professionally, and doing things in a clean and efficient way is frowned upon. Best take the weakest tool available and compensate for its shortcomings with third party services.
Since when trying to do better than using a tool designed for mass-producing software with dumb and cheap labor is "falling in love with ideals and tools"? It's kind of like saying that using an excavator over a bunch of people with shovels is "falling in love with ideals and tools". No, it just allows a single person to get more work done faster and better.
Our industry is weird. When recruiting, companies claim they want "the best of the best" and will often test you on ridiculous stuff. But when it comes to actual work, on industry scale, learning, growing professionally, and doing things in a clean and efficient way is frowned upon. Best take the weakest tool available and compensate for its shortcomings with third party services.
> trying to do better than using a tool designed for mass-producing software with dumb and cheap labor
This is completely disregarding the available developer pool of Java programmers, and the fact that a lot of them are really talented and can write efficient, production ready software.
The fact is, there is at least an order of magnitude more developers to choose from compared to Closure, and hiring and replacing people is hard. Ignoring business realities like this is bad, and akin to "falling in love with ideals and tools".
Also, someone using Java does not automatically make them dumb cheap programming labour, same as someone using Closure is not automatically going to be a 10xer who has memorised Knuth. It's possible to write bad FP code as well.
This is completely disregarding the available developer pool of Java programmers, and the fact that a lot of them are really talented and can write efficient, production ready software.
The fact is, there is at least an order of magnitude more developers to choose from compared to Closure, and hiring and replacing people is hard. Ignoring business realities like this is bad, and akin to "falling in love with ideals and tools".
Also, someone using Java does not automatically make them dumb cheap programming labour, same as someone using Closure is not automatically going to be a 10xer who has memorised Knuth. It's possible to write bad FP code as well.
> Also, someone using Java does not automatically make them dumb cheap programming labour, same as someone using Closure is not automatically going to be a 10xer who has memorised Knuth.
I'm not claiming that. I'm claiming that Java has a long history of enabling factory-farming software development, whereas Clojure doesn't lend itself to this style. I mean that in the sense that an army of cheap labor with shovels can, in principle, do the same job as a few trained operators in construction machines. The physics of dirt is the same, and so the job done is essentially the same - just some tools let you do it faster with fewer people. And then, the quantity (or lack of it) has a quality of its own - the more people you engage doing smaller and smaller pieces of work, the more your work becomes about coordinating people than doing the actual job.
(I'm aware that since version 8, Java is growing to be a quite decent programming language. This somewhat weakens my criticism, but not all that much.)
But yeah, I'm also claiming that ceteris paribus, the pool of Clojure developers will yield you higher-skilled programmers on average than the pool of Java programmers - simply because everyone and their dog knows some Java nowadays, and some Java is enough to make some progress in factory-farming software development, whereas Clojure is somewhat atypical and requires expanding your competences beyond the very basics.
(Or, in other words, I'm claiming that the distribution of skill of Java programmers is wider than that of Clojure programmers, with the latter having higher minimum and mean values, and comparable max values.)
I'm not claiming that. I'm claiming that Java has a long history of enabling factory-farming software development, whereas Clojure doesn't lend itself to this style. I mean that in the sense that an army of cheap labor with shovels can, in principle, do the same job as a few trained operators in construction machines. The physics of dirt is the same, and so the job done is essentially the same - just some tools let you do it faster with fewer people. And then, the quantity (or lack of it) has a quality of its own - the more people you engage doing smaller and smaller pieces of work, the more your work becomes about coordinating people than doing the actual job.
(I'm aware that since version 8, Java is growing to be a quite decent programming language. This somewhat weakens my criticism, but not all that much.)
But yeah, I'm also claiming that ceteris paribus, the pool of Clojure developers will yield you higher-skilled programmers on average than the pool of Java programmers - simply because everyone and their dog knows some Java nowadays, and some Java is enough to make some progress in factory-farming software development, whereas Clojure is somewhat atypical and requires expanding your competences beyond the very basics.
(Or, in other words, I'm claiming that the distribution of skill of Java programmers is wider than that of Clojure programmers, with the latter having higher minimum and mean values, and comparable max values.)
> I'm claiming that Java has a long history of enabling factory-farming software development, whereas Clojure doesn't lend itself to this style.
Any hugely popular language would have a lot of crap written in it, and a non-popular language wouldn't have "factory-farming" simply because it's not at the needed scale. So any non-mainstream language would almost by definition not lend itself to "factory-farming" development. You could also say that a lot of good stuff has been written in Java, and not as much good stuff has been written in Clojure, and conclude that Clojure doesn't lend itself to writing good stuff.
Any comparison between languages with at least 100x difference in popularity is meaningless, especially as no large bottom-line effect of programming languages has been found.
> the pool of Clojure developers will yield you higher-skilled programmers on average than the pool of Java programmers
Which matters if you're picking people from the pool at random. If not, you want to have the pool with the greater number of higher-skilled developers, not higher average. It's like saying that it makes more sense to start software companies in Finland than in the US, because on average people there are much better educated.
Any hugely popular language would have a lot of crap written in it, and a non-popular language wouldn't have "factory-farming" simply because it's not at the needed scale. So any non-mainstream language would almost by definition not lend itself to "factory-farming" development. You could also say that a lot of good stuff has been written in Java, and not as much good stuff has been written in Clojure, and conclude that Clojure doesn't lend itself to writing good stuff.
Any comparison between languages with at least 100x difference in popularity is meaningless, especially as no large bottom-line effect of programming languages has been found.
> the pool of Clojure developers will yield you higher-skilled programmers on average than the pool of Java programmers
Which matters if you're picking people from the pool at random. If not, you want to have the pool with the greater number of higher-skilled developers, not higher average. It's like saying that it makes more sense to start software companies in Finland than in the US, because on average people there are much better educated.
Does Clojure lend itself idiomatically to any style other than quick-and-dirty prototyping, though? There are real benefits to many features of Java when it comes to long-term maintainability, even though it also has some very real pitfalls at the same time (no optional values, no variant records with pattern matching, no composition-plus-delegation-over-inheritance etc. etc.).
Yes, it absolutely does. My team has projects that have been running for many years in production. They're far easier to maintain than equivalent Java projects we've done before. And there are plenty of companies, like Circle CI, who built there stack on Clojure out there. Here's a talk from Kira Systems who went from a quick and dirty startup to a 150 employee company in 8 years using Clojure: https://www.youtube.com/watch?v=u-4FiFpkPlQ
Considering Kira Systems had to resort to using Go for much of their heavy lifting, due to Clojure immutability memory overhead, I would hardly concur that this is a shining example of the benefits of using Clojure.
Over 80% of their codebase is in Clojure, but obviously it's providing them no benefit.
> non-results oriented
I think Clojure's ecosystem is more results-oriented than almost any other currently available[1]. It just requires a completely different mindset to understand, which an average software developer lacks. Therefore, Java is a much better fit for an Apache project.
[1] https://news.ycombinator.com/item?id=19728502
I think Clojure's ecosystem is more results-oriented than almost any other currently available[1]. It just requires a completely different mindset to understand, which an average software developer lacks. Therefore, Java is a much better fit for an Apache project.
[1] https://news.ycombinator.com/item?id=19728502
Oh I'm sure people make that argument all the time. I just don't think it holds up to reality. It's sort of just a dismissal of better options. The reality in my experience is that Java is just a poor choice for a lot of software. That's why you end up with frameworks built upon frameworks & AspectJ to basically make it behave like a dynamic language and try to hide some of the ugliness of the underlying APIs from you. And it's just because it's corporate approved and "everyone knows Java". I don't see how it can reasonably be considered the pragmatic choice. It's just not productive, and the final product isn't something that I would consider easy to understand or maintain most of the time either.
poor lisp, just like reddit ... always dropped for another language.
What if they are phases of evolution? A small team that can speak a lightweight language can quickly change velocity (speed and direction), iterating until a PoC is a useful system. Once the interfaces and the protocols fall into place, calcify or harden the design by converting to something more static, more literal.
Imagine going from something like lisp to Ada? Or lisp to Rust? Write quick prototypes in lisp, solve the hard problems of understanding the domain, then convert portions that require correctness to a language that is amenable to correctness checking (property testing, model checking, etc).
I think gradual typing [0] (not Gradual Typing) or language embedding [1] are steps in that direction. In a way, lisp can be thought of as a CASE tool for bootstrapping new systems.
The great thing about language platforms (JVM, Beam, Graal, Wasm, Racket) is that many languages can easily exist in the same system so that in many cases, code can be converted on a function by function basis.
[0] http://mypy-lang.org
[1] http://terralang.org
Imagine going from something like lisp to Ada? Or lisp to Rust? Write quick prototypes in lisp, solve the hard problems of understanding the domain, then convert portions that require correctness to a language that is amenable to correctness checking (property testing, model checking, etc).
I think gradual typing [0] (not Gradual Typing) or language embedding [1] are steps in that direction. In a way, lisp can be thought of as a CASE tool for bootstrapping new systems.
The great thing about language platforms (JVM, Beam, Graal, Wasm, Racket) is that many languages can easily exist in the same system so that in many cases, code can be converted on a function by function basis.
[0] http://mypy-lang.org
[1] http://terralang.org
The SW world is full of Java/C++ rewrites of products that originally started out in a dynamic & productive language like Lisp or Python. It might make sense in the lifecycle phase when you're past making big changes under make-or-break pressure, and are in the "provide predictable 10 year maintainance roadmap for this large codebaes" stage. You end up employing different kinds of programmers too because many of the original team may be reassigned to something more exciting/important to do.
But it's in no way inevitable, of course, there are many old Lisp, Python, etc codebases in production too.
BTW. I've been trying to rediscover this blog post and can't seem to find it again, anyone? It described different kinds of software projects, the (1) kind that try to deliver incremental improvements (eg 10% cost savings), and the (2) kind that try to make 10x improvements, and the difference in how those projects are run (the 1-type project can't go over budget because it will then deliver negative value etc).
But it's in no way inevitable, of course, there are many old Lisp, Python, etc codebases in production too.
BTW. I've been trying to rediscover this blog post and can't seem to find it again, anyone? It described different kinds of software projects, the (1) kind that try to deliver incremental improvements (eg 10% cost savings), and the (2) kind that try to make 10x improvements, and the difference in how those projects are run (the 1-type project can't go over budget because it will then deliver negative value etc).
Hopefully we'll start seeing some Rust rewrites in the future then, since the pitfalls of C++ are widely-known by now, and few people would claim that they don't impact long-term maintainability. (And the Rust Evangelism Strike Force would surely approve of such a shift!)
If Rust Evangelism Strike Force volunteers to rewrite code in Rust instead of heckling original developers to do it they may prove themselves to be more productive and useful.
also they can still have hybrid clojure jars if needs be
> While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.
It is interesting to contrast this with state of affairs in Apache Spark. Spark has thrived well in spite of being a Scala project; Scala arguably has a higher barrier for entry compared to Clojure (although the flavour of Scala used within Spark closely resembles Java).
It is interesting to contrast this with state of affairs in Apache Spark. Spark has thrived well in spite of being a Scala project; Scala arguably has a higher barrier for entry compared to Clojure (although the flavour of Scala used within Spark closely resembles Java).
In the old days a project would often start in Lisp (Common Lisp for example), and then later be rewritten in C.
This is the same pattern, and one should embrace it. Write the first cut, explore the problem space and prototype in a dynamic language. Once the design is proven, rewrite in statically typed language for performance and to get the possible remaining bugs out.
If the design was perfect from the get go, you could just write it in assembler to start with.
This is the same pattern, and one should embrace it. Write the first cut, explore the problem space and prototype in a dynamic language. Once the design is proven, rewrite in statically typed language for performance and to get the possible remaining bugs out.
If the design was perfect from the get go, you could just write it in assembler to start with.
For anyone interested in the Clojure history, there's this interview of Nathan Marz, originalcreator of Storm, from 5 years ago: https://www.infoq.com/interviews/marz-lambda-architecture/
While most comments touch on the developer issue, the significant performance gain deserves more discussion. I guess Java's static type, pragmatic mutability with less garbage produced, and efficient parallelism give substantial boost to performance.
It would be interesting to see what the upcoming blogs have to say about the performance gain.
It would be interesting to see what the upcoming blogs have to say about the performance gain.
My guess would be that it's always easier to make a rewrite performant because you already have a working version and know exactly where bottlenecks are. I imagine rewriting it in Clojure, or any other language, would've resulted in a significant performance boost as well.
Sad to see it happen, however it is yet another example how platform languages always win in the end, even if due to their nature and market size they tend to be more conservative regarding feature adoption.
I dare say it may be because they are more conservative regarding feature adoption.
> While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.
While I appreciate that programmers have limited free time and may not want to learn a new language to contribute to open source, I feel like learning Clojure could be of value to the programmer in itself.
The supposition that Clojure acts as a big barrier against contribution from developers might be true, but I really feel like developers who only write code in the ALGOL family of languages are missing out. Learning a lisp language is not an impossible task and can improve your skills as a developer.
While I appreciate that programmers have limited free time and may not want to learn a new language to contribute to open source, I feel like learning Clojure could be of value to the programmer in itself.
The supposition that Clojure acts as a big barrier against contribution from developers might be true, but I really feel like developers who only write code in the ALGOL family of languages are missing out. Learning a lisp language is not an impossible task and can improve your skills as a developer.
Can someone enlighten me on the inner working of Apache?
Do they hire developers to maintain and work on the projects? If not, is there community leads in charge of each one? How are they appointed?
Do they hire developers to maintain and work on the projects? If not, is there community leads in charge of each one? How are they appointed?
Interesting, where would you go to see the list of committers for a given project and find information about them?
I'm curious if the commiters are Clojure developers who chose to switch to Java, or if they are Java devs who decided to rewrite the project to a language they are more familiar and comfortable in. Or if they are Clojure devs looking to transition the project to other commiters and they struggled to find other Clojure devs willing to take it over.
I'm curious if the commiters are Clojure developers who chose to switch to Java, or if they are Java devs who decided to rewrite the project to a language they are more familiar and comfortable in. Or if they are Clojure devs looking to transition the project to other commiters and they struggled to find other Clojure devs willing to take it over.
We did the same with our open source project. We ditched Clojure and Clojurescript for NodeJS and JavaScript, for the same reason as it was a huge barrier to entry for new users
How am I just learning about this? This seems to be rather novel. Real-time bolt on for any database? Pretty cool stuff.
How does this relate to Apache Heron? I thought Heron was essentially Storm 2.0. Which one should I be using?
Not well known is that Heron's performance claims in the "flying faster" blog and SIGMOD paper were made by comparing against a fork of an older (pre apache) version of Storm even though much more performant version was already out.
IMO (biased?) Heron makes surprisingly poor design choices for their key distinguishing features like threading model and backpressure. And it shows when you benchmark it. More specifics: https://github.com/apache/storm/pull/2241#issuecomment-31787...
IMO (biased?) Heron makes surprisingly poor design choices for their key distinguishing features like threading model and backpressure. And it shows when you benchmark it. More specifics: https://github.com/apache/storm/pull/2241#issuecomment-31787...
Given there're so many streaming frameworks with different characteristics, you should be the one who understands the difference and select one fits for your business use case. There's no silver bullet.
If you're going to be on the JVM, stick to what the J stands for :)
The newest JVM out of Oracle, GraalVM, has its polyglossy as its central feature. And Clojure has always been a lot closer to Java (in terms of quality of interop) than even e.g. Scala, let alone something like Jython. (Notable exception JRuby.)
What exactly are you doing?
cutting off 1.0 immediately sounds quite harsh, I wonder what kind of user base it has?
"EOL for 1.0.x With the release of 2.0.0 the 1.0.x version line will no longer be maintained."
"EOL for 1.0.x With the release of 2.0.0 the 1.0.x version line will no longer be maintained."
> It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility.
Many big projects tend to manage around 3 version lines. Since Storm 2.0 is out, there're three version lines for 2.x, 1.2.x, 1.1.x. 1.0.x could be EOL-ed.
That's because they'll be maintaining the 1.1.x and 1.2.x branches...
DISCLAIMER: I'm one of PMC member of Apache Storm.
Even though Clojure has been Nathan's first-class language, not everyone in contributors and even committers couldn't become native on Clojure. (Though I guess part of committers could feel Clojure as native.) It might not be a problem when Nathan builds most of things in Storm (I'll link his blog post which describes lesson learned from Storm on his perspective http://nathanmarz.com/blog/history-of-apache-storm-and-lesso...), but when Storm becomes popular and also no longer his toy project, Clojure matters for scalibility of the project.
Personally, I had a hard time understanding plenty of macros and even chained macros - that might not be a problem if I were allowed to have plenty of times to study Clojure and get used to it, but possible contributors don't get a chance when they're ready. In many cases, the first time they read the code on what they're using is when they encounter issues on stage/production and have to dig with stack trace. For me, I had to investigate on Storm because my team suffered from lack of documentation and also lack of knowledge on internal which ended up just taking workarounds. It was pretty hard time learning language to just read and understand the details to track down the issue. I even couldn't think of writing code on Clojure.
When I decided to contribute Apache Storm, I started from connectors and CI build which wasn't written on Clojure. After some months I was able to read and understand the code to get into details but I still couldn't be native for Clojure, so still required N times of efforts to read and write the code even after I became one of PMC member.
Don't get me wrong. I'm not saying something is better than some other one. In scalability point of view on project, I guess we had to do this, even though it ended up teaching me some lesson that migrating language is not the thing which can be considered easily, especially for a big project. Lots of efforts were spent here.
Someone could claim for Apache Spark's case for Scala. Yes, I'm also contributing to Apache Spark and I had to learn Scala (fortunately I had a chance to do that before contributing) but the case is different. If you read Databricks Scala guide in details, you might realize that the doc is actually not encouraging contributors to use advanced features (a.k.a the features only Scala geeks might feel natural) on Scala. Same goes on reading codebase. Apache Spark community has been trying to keep necessary knowledges on Scala to contribute Apache Spark low enough, so that non-expert of Scala engineer could easily try out contribution. If they have been requiring hard understanding of Scala to accept contributions - Spark may not be able to get its amazing reputation.
Even though Clojure has been Nathan's first-class language, not everyone in contributors and even committers couldn't become native on Clojure. (Though I guess part of committers could feel Clojure as native.) It might not be a problem when Nathan builds most of things in Storm (I'll link his blog post which describes lesson learned from Storm on his perspective http://nathanmarz.com/blog/history-of-apache-storm-and-lesso...), but when Storm becomes popular and also no longer his toy project, Clojure matters for scalibility of the project.
Personally, I had a hard time understanding plenty of macros and even chained macros - that might not be a problem if I were allowed to have plenty of times to study Clojure and get used to it, but possible contributors don't get a chance when they're ready. In many cases, the first time they read the code on what they're using is when they encounter issues on stage/production and have to dig with stack trace. For me, I had to investigate on Storm because my team suffered from lack of documentation and also lack of knowledge on internal which ended up just taking workarounds. It was pretty hard time learning language to just read and understand the details to track down the issue. I even couldn't think of writing code on Clojure.
When I decided to contribute Apache Storm, I started from connectors and CI build which wasn't written on Clojure. After some months I was able to read and understand the code to get into details but I still couldn't be native for Clojure, so still required N times of efforts to read and write the code even after I became one of PMC member.
Don't get me wrong. I'm not saying something is better than some other one. In scalability point of view on project, I guess we had to do this, even though it ended up teaching me some lesson that migrating language is not the thing which can be considered easily, especially for a big project. Lots of efforts were spent here.
Someone could claim for Apache Spark's case for Scala. Yes, I'm also contributing to Apache Spark and I had to learn Scala (fortunately I had a chance to do that before contributing) but the case is different. If you read Databricks Scala guide in details, you might realize that the doc is actually not encouraging contributors to use advanced features (a.k.a the features only Scala geeks might feel natural) on Scala. Same goes on reading codebase. Apache Spark community has been trying to keep necessary knowledges on Scala to contribute Apache Spark low enough, so that non-expert of Scala engineer could easily try out contribution. If they have been requiring hard understanding of Scala to accept contributions - Spark may not be able to get its amazing reputation.
Why change the title of the post?
Almost every one of the ~100 comments is in response to the original title that Storm switched from Clojure to Java. Now this context is lost.
Almost every one of the ~100 comments is in response to the original title that Storm switched from Clojure to Java. Now this context is lost.
Because the previous title was editorialized? Furthermore, perhaps the conversation is about the old title because it was, and now the conversation can go be about Storm itself :)
I think the rule about not editorializing needs an exception so that someone can point out an important or interesting fact using an article whose original title is too generic.
Another example would be "Press release from XYZ". If it talks about XYZ reaching some major breakthrough, the title should be allowed to reflect that. It's a waste of everyone's time to post the article titled "Press release".
And arguably this situation should fall under such an exception.
Another example would be "Press release from XYZ". If it talks about XYZ reaching some major breakthrough, the title should be allowed to reflect that. It's a waste of everyone's time to post the article titled "Press release".
And arguably this situation should fall under such an exception.
Java is statically typed. Clojure is not. To me, this is the biggest difference and the reason to choose Java.
In the critical path many things can matter that ordinarily don't. When working on perf improvements for Storm 1.0 release, profiling revealed hotspots in Clojure code. Adding type hints to avoid dynamic typing fixed some of these hotspots. In other places, Clojure primitives showed up as a bottleneck. These required rewriting some code in Java and invoking it from Clojure.
This was mentioned briefly here: https://hortonworks.com/blog/microbenchmarking-storm-1-0-per...
The Clojure based core has served the community well for some time. It was still powered by Disruptor (a fast Java messaging library). So there was always some back and forth between Java and Clojure.
For Storm 2.0, the Java rewrite happened first. The re-architecture work might have not happened without it (at least for the foreseeable near future)... primarily due to Java expertise being more easily available.
This was mentioned briefly here: https://hortonworks.com/blog/microbenchmarking-storm-1-0-per...
The Clojure based core has served the community well for some time. It was still powered by Disruptor (a fast Java messaging library). So there was always some back and forth between Java and Clojure.
For Storm 2.0, the Java rewrite happened first. The re-architecture work might have not happened without it (at least for the foreseeable near future)... primarily due to Java expertise being more easily available.
[deleted]
Is this true or false? I don't understand the downvotes.
While it is true that Java is statically typed and Clojure is dynamically typed it is perhaps one of the less important differences between the two languages. If you in fact would like to add a static type system on top of Clojure there is a way to do that: Typed Clojure.
https://typedclojure.org/
https://typedclojure.org/
I'm not going to downvote the comment, but I think the "statically typed" argument is fantastically overrated. I spent years of my life coding in Java and C# and other years of my life writing Clojure, Python, Ruby, and Elixir code. I've never once thought to myself of the latter, "Damn! This would be so much better if I could have static types." It has literally never, not once, ever been an issue for me.
Many people disagree with "the reason to choose Java".
While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.
Very common story that I hear from a lot of shops that tried to go Clojure first. Great language, but too little penetration amongst developers. Makes it very difficult to effectively recruit employees/contributors.
At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.