Macros: the Plan for Scala 3(scala-lang.org)
scala-lang.org
Macros: the Plan for Scala 3
https://www.scala-lang.org/blog/2018/04/30/in-a-nutshell.html
30 comments
> Unpopular opinion: I hate meta programming.
As papaf asked, is it metaprogramming you hate, or bad metaprogramming? Metaprogramming is just another tool in the toolbox, and hating syntactic abstraction (macros) makes as much sense to me as hating procedural abstraction (functions). Yes, it’s easy to write bad syntactic abstractions, but it’s also very easy to write hilariously bad procedural abstractions, too.
cbzbc states that metaprogramming is often harder to get write than functions. I’m not certain that’s true; might it just not be the case that most programmers just don’t get very much experience with metaprogramming, since they’re typically using languages with very underdeveloped facilities for it?
> A long time ago, I was freelancing Rails. I can't accurately describe the deflating feeling of tracing your problem back to some class that one person (a ruby guru warrior, no doubt) wrote, with a few dozen dynamic deeds or to_methods. It's the most demotivating thing I've come across in real life code.
My own experience is that for some reason Ruby tends to attract programmers who enjoy doing things because they can be done, not because they should be. It’s so bad that my general rule is not to use systems programmed in Ruby, because I’ve been burnt so many times before. That’s not entirely fair (obviously there are some great systems written in Ruby — e.g. I believe that both GitHub & GitLab are), but it reflects my experience.
As papaf asked, is it metaprogramming you hate, or bad metaprogramming? Metaprogramming is just another tool in the toolbox, and hating syntactic abstraction (macros) makes as much sense to me as hating procedural abstraction (functions). Yes, it’s easy to write bad syntactic abstractions, but it’s also very easy to write hilariously bad procedural abstractions, too.
cbzbc states that metaprogramming is often harder to get write than functions. I’m not certain that’s true; might it just not be the case that most programmers just don’t get very much experience with metaprogramming, since they’re typically using languages with very underdeveloped facilities for it?
> A long time ago, I was freelancing Rails. I can't accurately describe the deflating feeling of tracing your problem back to some class that one person (a ruby guru warrior, no doubt) wrote, with a few dozen dynamic deeds or to_methods. It's the most demotivating thing I've come across in real life code.
My own experience is that for some reason Ruby tends to attract programmers who enjoy doing things because they can be done, not because they should be. It’s so bad that my general rule is not to use systems programmed in Ruby, because I’ve been burnt so many times before. That’s not entirely fair (obviously there are some great systems written in Ruby — e.g. I believe that both GitHub & GitLab are), but it reflects my experience.
I think the challenge with "metaprogramming" is coming up with a good debugging model.
The strength of functional programming is the ease of reasoning about the computation model. "This function is called with these arguments, and I can see it will then always return this exact result." Imperative programming makes predicting what a program will do much, much more difficult. Then metaprogramming can take understanding a program's behavior up another level of difficulty.
I experience this programming in Java with Annotation Oriented Programming (the preferred model of Spring). I find it even more difficult than Ruby "define method" or Lisp macros to deduce the behavior of a program. It's really hard to go from seeing an annotation in the program, to figuring out how exactly the annotation will change the program's behavior. In practice, it's pretty much just "Stack Overflow and Pray".
The strength of functional programming is the ease of reasoning about the computation model. "This function is called with these arguments, and I can see it will then always return this exact result." Imperative programming makes predicting what a program will do much, much more difficult. Then metaprogramming can take understanding a program's behavior up another level of difficulty.
I experience this programming in Java with Annotation Oriented Programming (the preferred model of Spring). I find it even more difficult than Ruby "define method" or Lisp macros to deduce the behavior of a program. It's really hard to go from seeing an annotation in the program, to figuring out how exactly the annotation will change the program's behavior. In practice, it's pretty much just "Stack Overflow and Pray".
> hating syntactic abstraction (macros) makes as much sense to me as hating procedural abstraction (functions).
This is a bit pedantic, but...
Many people's experiences with macros are through C. C macros are not syntactic abstraction, but some sort of lexical expansion that introduced syntactic confusion.
I get this is not the case in many languages now but for most popular languages macros are "throw appended strings into eval/C preprocessors"
This is a bit pedantic, but...
Many people's experiences with macros are through C. C macros are not syntactic abstraction, but some sort of lexical expansion that introduced syntactic confusion.
I get this is not the case in many languages now but for most popular languages macros are "throw appended strings into eval/C preprocessors"
I agree!
I suppose that the wish for meta programming, when I rely on it, is to express a pattern that either can't express with types or with functions. In ruby, I know well enough to avoid define method and mm, and do things the hard way, but in other languages the temptations are still there.
My first foray into programming was Scheme, through SICP. So, I was exposed to a lot of meta programming early on, but I wasn't adept and certainly the majority of that book is beyond what I can comprehend.
I love functional programming, and especially a language with a mature type system, because it holds my expressions to a standard that alleviates that urge to use a dynamic def, and a wackedy-hackedy solution.
I hate that there are difficult concepts that can't be easily expressed without syntax transforms, and that so many of these macro systems make code harder to read.
Honestly, even scheme's macros are hard to read at first. You're thinking about three entire levels of computation, the forms, where its evaluating, and where the continuation is, it's completely unmanageable, or so I think. Other people seem to have an easier time of it.
Am I a bad programmer for not being able to metaprogram well? Certainly some of these systems are bad, and certainly some of the blame for my dislike of them is within myself also. But I don't think I'm a bad programmer for wanting to express my code in the most compositional, logical, functional, as close to unambiguous manner as possible, and to that end I avoid macros when I can.
I also dread seeing them, especially someone else's code that I'm gonna have to debug or refactor, and then I was the last person to touch it so when it nearly inevitably blows up, it's magically my fault.
I don't think the merit of syntax transforms (macros) and value transforms(functions) are equal.
In a situation where the macro is typesafe and obeys black box principle, they're essentially equivalent in the manner of their side effects, but in the vast majjority of real world situations, that's not the case. Rails builds take forever, and early on, it was widely accepted that a rails app would crash dozens of times a day, and cutting edge rails convention was forged by people figuring out how to start it back up quickly.
I can imagine there are some CL defmacro horror stories floating around out there.
Hell, LtU was figuring out how to cheat syntax-rules years ago.
I hope this brings you some sort of insight. Sorry for this mess of paragraphs. I'm generally not allowed off of my spreadsheets, so I'm just a dorky hermit, and I can't express my thoughts into the right words, really.
Anyways, yes. I guess I hate meta programming and prefer higher order FP.
I suppose that the wish for meta programming, when I rely on it, is to express a pattern that either can't express with types or with functions. In ruby, I know well enough to avoid define method and mm, and do things the hard way, but in other languages the temptations are still there.
My first foray into programming was Scheme, through SICP. So, I was exposed to a lot of meta programming early on, but I wasn't adept and certainly the majority of that book is beyond what I can comprehend.
I love functional programming, and especially a language with a mature type system, because it holds my expressions to a standard that alleviates that urge to use a dynamic def, and a wackedy-hackedy solution.
I hate that there are difficult concepts that can't be easily expressed without syntax transforms, and that so many of these macro systems make code harder to read.
Honestly, even scheme's macros are hard to read at first. You're thinking about three entire levels of computation, the forms, where its evaluating, and where the continuation is, it's completely unmanageable, or so I think. Other people seem to have an easier time of it.
Am I a bad programmer for not being able to metaprogram well? Certainly some of these systems are bad, and certainly some of the blame for my dislike of them is within myself also. But I don't think I'm a bad programmer for wanting to express my code in the most compositional, logical, functional, as close to unambiguous manner as possible, and to that end I avoid macros when I can.
I also dread seeing them, especially someone else's code that I'm gonna have to debug or refactor, and then I was the last person to touch it so when it nearly inevitably blows up, it's magically my fault.
I don't think the merit of syntax transforms (macros) and value transforms(functions) are equal.
In a situation where the macro is typesafe and obeys black box principle, they're essentially equivalent in the manner of their side effects, but in the vast majjority of real world situations, that's not the case. Rails builds take forever, and early on, it was widely accepted that a rails app would crash dozens of times a day, and cutting edge rails convention was forged by people figuring out how to start it back up quickly.
I can imagine there are some CL defmacro horror stories floating around out there.
Hell, LtU was figuring out how to cheat syntax-rules years ago.
I hope this brings you some sort of insight. Sorry for this mess of paragraphs. I'm generally not allowed off of my spreadsheets, so I'm just a dorky hermit, and I can't express my thoughts into the right words, really.
Anyways, yes. I guess I hate meta programming and prefer higher order FP.
> Unpopular opinion: I hate meta programming
Come to the dark side and start programming Clojure. Because there, meta programming is no different than regular programming -- your macros are just normal Clojure functions that happen to run at compile time, and they have the full language to do whatever you want.
Come to the dark side and start programming Clojure. Because there, meta programming is no different than regular programming -- your macros are just normal Clojure functions that happen to run at compile time, and they have the full language to do whatever you want.
Even in Clojure, you probably shouldn’t if there’s an alternative. Full disclosure: I have written useful Clojure macros.
Functions are ultimately more powerful since you can store them, pass them to other functions, partially apply them, etc. You can't do that with macros, of course.
But macros allow very repetitive stuff to be automated which is a real productivity win.
But macros allow very repetitive stuff to be automated which is a real productivity win.
But there's no types! I've never tried clojure, but I've done scheme before. Maybe I should... Thanks!
Li Haoyi shared his alternative to sbt here a little while ago. Looks good to me.
http://www.lihaoyi.com/mill/
http://www.lihaoyi.com/mill/
Not a Scala developer, but...as a JVM language, what's stopping Scala from just adopting Gradle or Maven as a build tool? Maybe with just a thin sheen of Scala syntax on top of the underlying model?
As a working Java programmer, the build tools are one of the more pleasant parts of the development experience. I do some Ruby development at home (Ruby Motion, actually), and the build and dependency management tools seem pretty kludgy and patched together in comparison.
As a working Java programmer, the build tools are one of the more pleasant parts of the development experience. I do some Ruby development at home (Ruby Motion, actually), and the build and dependency management tools seem pretty kludgy and patched together in comparison.
Nothing - I have seen projects using Maven and Gradle, working well.
Convention, mainly.
I've seen it a while ago too, but without lightbend blessing it's next to useless, in practical usage. I can't deliver that to a client
Stripe uses Bazel to build Scala code: https://github.com/bazelbuild/bazel/wiki/Bazel-Users#stripe
Other large companies use Gradle or even Ant to build Scala code.
Why would a client be restricted to sbt?
Other large companies use Gradle or even Ant to build Scala code.
Why would a client be restricted to sbt?
Because SBT is (unfortunately) the standard build tool for Scala.
So if you're trying to do any cross project compilation to Scala Native or ScalaJS then there is no alternative to SBT.
Likewise many much needed Scala build tools only work with SBT.
So if you're trying to do any cross project compilation to Scala Native or ScalaJS then there is no alternative to SBT.
Likewise many much needed Scala build tools only work with SBT.
There is more to it, sbt has a very rich echo system of plugins, it will take time for any alternative to gain traction.
Of course, I agree. In time, as the scala ecosystem flourishes and adapts, I, along with many others, may be coaxed back in.
Scala suffers from a problem of too many bright people all trying to solve too few problems. And implicits. Bad idea there. But the rest of it is logistical, and just takes time, as you said.
Have faith! Learn ocaml or f# too, but have faith all the same.
Scala suffers from a problem of too many bright people all trying to solve too few problems. And implicits. Bad idea there. But the rest of it is logistical, and just takes time, as you said.
Have faith! Learn ocaml or f# too, but have faith all the same.
> Scala suffers from a problem of too many bright people all trying to solve too few problems
1000x yes.
And bright people who are so bright that they simply can't appreciate what the user experience is for people who aren't bright. These people are from a lot of experience a bit toxic. And it's a bad choice of word and it isn't personal but because they are on a different level technically they can really change teams for the worse.
At some point Scala is going to need to better reconcile the Typelevel world of Scala and the Play/Java world of Scala. Otherwise languages like Kotlin is going to chip away until nothing is left.
1000x yes.
And bright people who are so bright that they simply can't appreciate what the user experience is for people who aren't bright. These people are from a lot of experience a bit toxic. And it's a bad choice of word and it isn't personal but because they are on a different level technically they can really change teams for the worse.
At some point Scala is going to need to better reconcile the Typelevel world of Scala and the Play/Java world of Scala. Otherwise languages like Kotlin is going to chip away until nothing is left.
I hate meta programming. We have composition, why do you need to mess with the nature of things?
Maybe you just hate bad metaprogramming? In theory metaprogramming makes a language more powerful, leads to less boilerplate code, and can deliver big performance improvements.
Scala gives programmers plenty of power which comes with the tradeoffs that there is plenty of opportunity to shoot yourself in the foot or produce unmaintainable code. Scala makes different tradeoffs to Go.
Personally, I think that ignoring metaprogramming means missing out on one of the most powerful techniques available to programmers.
Maybe you just hate bad metaprogramming? In theory metaprogramming makes a language more powerful, leads to less boilerplate code, and can deliver big performance improvements.
Scala gives programmers plenty of power which comes with the tradeoffs that there is plenty of opportunity to shoot yourself in the foot or produce unmaintainable code. Scala makes different tradeoffs to Go.
Personally, I think that ignoring metaprogramming means missing out on one of the most powerful techniques available to programmers.
Maybe you just hate bad metaprogramming
Metaprogramming is hard to get right - often harder than writing good libraries, and the number of programmers who can do it well is correspondingly smaller.
So of the metaprogramming that's out there in the wild, a correspondingly greater proportion of it is bad.
Metaprogramming is hard to get right - often harder than writing good libraries, and the number of programmers who can do it well is correspondingly smaller.
So of the metaprogramming that's out there in the wild, a correspondingly greater proportion of it is bad.
Rule 1 of metaprogramming: don't use it when something else will do (almost) as well.
Still, I think there is a need. Take declarative DSLs for instance. The problem with how that's done in all languages right now is that these DSL are running code that build a tree model of sorts.
In some languages like Java or C# that's already ugly because the code must live in an variable initializer or method of a class somewhere (why couldn't it be it's own thing?).
Another problem: how to "refer" to a different part of your model? Solution 1: pass bits of trees around. So you need to store them in variables. In almost all languages this precludes forward references and circular structures. Solution 2: use strings, with all the safety and performance issues that does entail.
Then you have to realize the model is not first-class, it's just a data structure being built. So you can't generate useful methods based on it - you need to have generic functions that take your model as input and interpret it on the fly.
There are many many more use cases, but this one is the one that irks me the most. Why the hell do I need sinful XML, Groovy or what not to compile my Java project. Heck, even make. With some decent (compile-time!) meta-programming support, I could have all this as a DSL with all the might of the language tooling (autocompletion, IDEs etc) to bear.
Still, I think there is a need. Take declarative DSLs for instance. The problem with how that's done in all languages right now is that these DSL are running code that build a tree model of sorts.
In some languages like Java or C# that's already ugly because the code must live in an variable initializer or method of a class somewhere (why couldn't it be it's own thing?).
Another problem: how to "refer" to a different part of your model? Solution 1: pass bits of trees around. So you need to store them in variables. In almost all languages this precludes forward references and circular structures. Solution 2: use strings, with all the safety and performance issues that does entail.
Then you have to realize the model is not first-class, it's just a data structure being built. So you can't generate useful methods based on it - you need to have generic functions that take your model as input and interpret it on the fly.
There are many many more use cases, but this one is the one that irks me the most. Why the hell do I need sinful XML, Groovy or what not to compile my Java project. Heck, even make. With some decent (compile-time!) meta-programming support, I could have all this as a DSL with all the might of the language tooling (autocompletion, IDEs etc) to bear.
This post, as far as I can see, does not mention scala-meta-based macros (project called scalamacros I think) at all. What happened with this direction of macros development? Was it abandoned? Or is the new approach built on it?
It seems that scala-meta is alive (http://scalameta.org), but is now about developer tools (e.g. code formatting plugin used by IntelliJ). The macro part of scala-meta was spun off to https://github.com/scalacenter/macros, which is archived and points to Dotty macros as the way forward.
I'll be watching Dotty macros with keen interest, as I have found the current scala macro system very useful despite it remaining experimental. One thing I couldn't do with current macros was provide any kind of generic wrapper around macro methods in the same library, due to the separate compilation pass requirement. I hope this is what staging will address.
I'll be watching Dotty macros with keen interest, as I have found the current scala macro system very useful despite it remaining experimental. One thing I couldn't do with current macros was provide any kind of generic wrapper around macro methods in the same library, due to the separate compilation pass requirement. I hope this is what staging will address.
Prompted by a similar question on Reddit, I have written a comment on the history, the scope and the status of Scalameta: https://www.reddit.com/r/scala/comments/8g359q/macros_the_pl....
Hopefully this will simplify Scala for newcomers and especially speed up onboarding when handing over codebases written by expert developers.
Unpopular opinion: I hate meta programming. We have composition, why do you need to mess with the nature of things? A long time ago, I was freelancing Rails. I can't accurately describe the deflating feeling of tracing your problem back to some class that one person (a ruby guru warrior, no doubt) wrote, with a few dozen dynamic deeds or to_methods. It's the most demotivating thing I've come across in real life code.
It's worse somehow than complicated numeric code, because with the numbers, at least I know this shit works by the end of it. If I start writing good var names, I'll probably even figure out how it works in an hour or two
That being said, I'm tentatively excited about Tasty. What a witty name by the way!
Having the extra type information should allow for much smarter transforms at the language level and I hope macros in scala 3 aren't so much a mess.
Also, let's just get rid of implicits, whaddaya say? ;-)
Anyways, proud of the scala 3 team this far. Even though they haven't killed sbt yet, they're putting their work in admirably to a difficult problem. I'd love to have that much meaning and purpose in my own work.
I didn't have time to actually read spec, will do later, so if I said something inaccurate, let me know gently
Edit: macros will conform to black box and be constrained to their types, runs after type checking.
That's awesome