Why Crystal is the most promising programming language of 2018(medium.com)
medium.com
Why Crystal is the most promising programming language of 2018
https://medium.com/@DuroSoft/why-crystal-is-the-most-promising-programming-language-of-2018-aad669d8344f
108 comments
The number of open issues is a bad correlation for how well a project is managed: rust has nearly 4000 open issues. Most of these are either feature requests we'll get to some day, or minor bugs. Crystal development is proceding nicely and we will have two developers (bcardiff and me) working full-time on crystal this summer.
That's great and thank you for responding!
I agree that the number of open issues is not a good indicator of project management. I was more worried about PRs being open in 2016, 2017 and still lingering - this in connection with a large number of open issues seemed like a possible red flag. But it may be just an issue of triage.
Keep up the good work!
Keep up the good work!
Every project has it's warts, long running feature requests, and it's bugs so minor nobody's gotten around to fixing them.
Some projects decide to hide those by closing the issues, we generally keep them open, because they are still valid issues.
Some projects decide to hide those by closing the issues, we generally keep them open, because they are still valid issues.
And I think this is a good sign of a projects positive health ^
I for one have found the crystal community to be very responsive to the issues I have reported. I'd rather have a project leave 500 issues open than just close everything in sight like many large projects do.
>I'm not sure if this is a sign of poor project managment, development stall or poor community engagement
You missed another possibility: it being perfectly normal.
If you check any large-ish project you'll find 100s or 1000s of unsolved issues and stale PRs.
And I'm talking about seasoned, used in production, products, from MySQL to Chromium and whatever...
You missed another possibility: it being perfectly normal.
If you check any large-ish project you'll find 100s or 1000s of unsolved issues and stale PRs.
And I'm talking about seasoned, used in production, products, from MySQL to Chromium and whatever...
I think issues left open > 12 months is antithetical to the idea of lean/agile and could be a sign of suboptimal project management.
But don’t get me wrong, I <3 Ruby and am excited about Crystal.
But don’t get me wrong, I <3 Ruby and am excited about Crystal.
> I think issues left open > 12 months is antithetical to the idea of lean/agile and could be a sign of suboptimal project management.
Totally depends on the nature of the project, its policies and users.
Any popular FOSS project that allows anyone to post a bug, and doesn't just close unattended bug reports (which is bad imho), is likely to end up with some (usually minor) bugs open over 12 or more months.
Totally depends on the nature of the project, its policies and users.
Any popular FOSS project that allows anyone to post a bug, and doesn't just close unattended bug reports (which is bad imho), is likely to end up with some (usually minor) bugs open over 12 or more months.
>issues left open > 12 months is anti-thetical to the idea of lean/agile
OSS doesn't care about lean/agile, those are ideologies consultants sell to enterprises.
Things are done when they are done.
OSS doesn't care about lean/agile, those are ideologies consultants sell to enterprises.
Things are done when they are done.
> Nothing else puts all of these ingredients together (compile-time macros, static typing, C-like speed, Ruby-like syntax, gem-like package ecosystem, native binary compilation, fibers, and cross-platform support)
Something I've noticed when reading 'X language is great' posts, the comparison is usually limited to a narrow subset of what's out there.
As an example, I think D meets all of the above criteria. Maybe I'll give on Ruby-like syntax, which D instead has more of a C-like syntax with adjustments to improve compile time.
There are lots of great languages out there and it really irritates me when I see claims of 'X is better than Y or Z' and it seems no effort is made to see what else already exists with the same (or greater) set of features.
Something I've noticed when reading 'X language is great' posts, the comparison is usually limited to a narrow subset of what's out there.
As an example, I think D meets all of the above criteria. Maybe I'll give on Ruby-like syntax, which D instead has more of a C-like syntax with adjustments to improve compile time.
There are lots of great languages out there and it really irritates me when I see claims of 'X is better than Y or Z' and it seems no effort is made to see what else already exists with the same (or greater) set of features.
Native binary is not a clear statement. On Android is CPU-native, really native? Or is CLR the future of Windows? CPU-native does not mean much. It means that you have a competition of almost all languages, that have head start and better tooling.
"Is FFI cheap on all interesting platforms?" is a better question, than "Does it compile to CPU-native binary on all interesting platforms?". In this space you only kinda get some JVM languages, but on Windows you would have to do something more special. Scheme to some extent. Also Haxe.
If you don't care about it you can read the rest of this thread with all other favorite languages. I for one cheer for Myrddin, Pony and Zig.
I was thinking lately about an imperative language, that would be easily compiled to other first class languages. To serve as a cross-platform core of application. It would be quite limited - for example no heap allocations. Something maybe akin to Google's Wuffs language. Just an idea on the one edge of the problem.
"Is FFI cheap on all interesting platforms?" is a better question, than "Does it compile to CPU-native binary on all interesting platforms?". In this space you only kinda get some JVM languages, but on Windows you would have to do something more special. Scheme to some extent. Also Haxe.
If you don't care about it you can read the rest of this thread with all other favorite languages. I for one cheer for Myrddin, Pony and Zig.
I was thinking lately about an imperative language, that would be easily compiled to other first class languages. To serve as a cross-platform core of application. It would be quite limited - for example no heap allocations. Something maybe akin to Google's Wuffs language. Just an idea on the one edge of the problem.
Not only that, while ongoing Crystal efforts might be great, lets not forget it is a path already traced by the likes of Common Lisp, Dylan and Ruby motion.
So other that the gem-like package ecosystem and a syntax familiar to Ruby developers, it isn't particularly new.
So other that the gem-like package ecosystem and a syntax familiar to Ruby developers, it isn't particularly new.
This is the entire reason I got burned out on new languages.
I learned to minimize burns by looking at everything that comes out as learning process, there is always something that makes us better developers.
However for production code I only use the programming languages directly supported by platform owners.
Of course others rather take part in ramping up eco-systems and that is fine as well, otherwise we wouldn't get new toys adopted by platform owners.
However for production code I only use the programming languages directly supported by platform owners.
Of course others rather take part in ramping up eco-systems and that is fine as well, otherwise we wouldn't get new toys adopted by platform owners.
True. I guess I mean that after a while I started classifying languages like this as "flavors of C". There are just so many languages that are fast, it's no longer a selling point.
Out of all of the languages I've read about the past few years, only 3 jumped out as bringing real value to the table compared to the other options that are already out there: Go, Rust and Elixir.
Out of all of the languages I've read about the past few years, only 3 jumped out as bringing real value to the table compared to the other options that are already out there: Go, Rust and Elixir.
Could you elaborate about the value of Elixir, please?
It's been on my radar for quite a while now.
So a lot has been written on this topic already. I would personally direct you to read this (on Erlang, but Elixir is basically Erlang with nicer syntax and a few added bits) - https://ferd.ca/the-zen-of-erlang.html
Fundamentally though, Erlang/Elixir is a language built from first principles to achieve reliability in concurrent, distributed contexts. It's quite unique in that regard. Even the language it's most directly compared to (Go, in having go channels that are sort of but not quite like Erlang/Elixir processes) is built instead from concurrency first, not reliability (and having concurrency flow out from that as a required feature).
Fundamentally though, Erlang/Elixir is a language built from first principles to achieve reliability in concurrent, distributed contexts. It's quite unique in that regard. Even the language it's most directly compared to (Go, in having go channels that are sort of but not quite like Erlang/Elixir processes) is built instead from concurrency first, not reliability (and having concurrency flow out from that as a required feature).
Well, here is why not. AFAK, it still lacks support for parallelism / multicore programming. Please correct me if I'm wrong.
Once it has that, I'll be happy to learn it, but currently it's concurrency seems to be as limited as Racket's. That's a big obstacle for early adoption, since the trend is going towards 64 cores sooner than later. My current CPU already has 8 physical cores and 16 logical cores, and I'd like to use all of them.
This is not a critique on Crystal or Racket, I understand perfectly well the challenges that parallelism poses to language implementers. But it's definitely a massive shortcoming.
Once it has that, I'll be happy to learn it, but currently it's concurrency seems to be as limited as Racket's. That's a big obstacle for early adoption, since the trend is going towards 64 cores sooner than later. My current CPU already has 8 physical cores and 16 logical cores, and I'd like to use all of them.
This is not a critique on Crystal or Racket, I understand perfectly well the challenges that parallelism poses to language implementers. But it's definitely a massive shortcoming.
This is a big advantage of Nim (https://nim-lang.org/), which on the face of it is a pretty similar kind of language with a more Python-like syntax. Nim supports real threads with message passing channels (similar to Go).
> Nim supports real threads with message passing channels (similar to Go).
Could you specify what are the differences from Go, in this regard are, please?
Could you specify what are the differences from Go, in this regard are, please?
Sure - I'm no expert in concurrency in programming languages as such but here's my understanding: so Nim supports what are essentially native operating system threads but with a difference from most languages being that each thread has its own garbage collected heap. Nim supports shared memory if you so choose to use it with a shared heap. Nim also supports a channel based communication system similar to Go's.
Nim also supports lighter-weight parallelism with the `spawn` (unsafe, unstructured parallelism which schedules a procedure to run in a thread pool) and `parallel` (lets you use a subset of the Nim language to create concurrency safe tasks) keywords which pass tasks to thread pools in the background. You can read more about Nim's concurrency here: https://nim-lang.org/docs/manual.html#parallel-spawn-spawn-s...
Go has Goroutines which use less resources than threads on most operating systems while also taking advantage of multicore CPUs (I assume they must run in a thread pool running in the background?). It also supports a select case statement which is a nice bit of syntax sugar for managing multiple channels, which Nim doesn't support (although its macro system is pretty powerful, so you could plausibly come up with an equivalent). Go does not support shared memory communication as a language feature, although some libraries apparently give you that option.
Nim also supports lighter-weight parallelism with the `spawn` (unsafe, unstructured parallelism which schedules a procedure to run in a thread pool) and `parallel` (lets you use a subset of the Nim language to create concurrency safe tasks) keywords which pass tasks to thread pools in the background. You can read more about Nim's concurrency here: https://nim-lang.org/docs/manual.html#parallel-spawn-spawn-s...
Go has Goroutines which use less resources than threads on most operating systems while also taking advantage of multicore CPUs (I assume they must run in a thread pool running in the background?). It also supports a select case statement which is a nice bit of syntax sugar for managing multiple channels, which Nim doesn't support (although its macro system is pretty powerful, so you could plausibly come up with an equivalent). Go does not support shared memory communication as a language feature, although some libraries apparently give you that option.
All the goroutines share the heap, so go does support shared memory, it's just you're encouraged to use channels instead because they're easier to get right.
Crystal has the foundation for parallelism, it has the same concurrency model as go. Adding parallelism will happen before 1.0. Adding parallelism to crystal is easy, it's already been done in a branch and it works fine. The problem is getting the implementation fast enough that it doesn't regress performance for IO bound workloads. It's coming, just wait :)
> AFAK, it still lacks support for parallelism / multicore programming.
Yeah this is a deal-breaker. And it's kinda weird that the article is pitching Crystal as a replacement for Go, Elixir, and Rust with that kind of limitation.
I guess this is just the ramp-up into one more trip around the giant ferris-wheel that is language popularity cycles.
Yeah this is a deal-breaker. And it's kinda weird that the article is pitching Crystal as a replacement for Go, Elixir, and Rust with that kind of limitation.
I guess this is just the ramp-up into one more trip around the giant ferris-wheel that is language popularity cycles.
Are you sure Racket doesn't support parallelism?
Racket's support for parallelism with futures and places is in my experience almost unusable. I don't understand why they have chosen these horrible abstractions rather than providing access to OS-level threads plus some reasonable synchronization mechanisms like channels, shared memory, or actors.
Before you criticize me for this statement, please let me assure you that it's based on plenty of experience. I'm using Racket for almost all my programming since Racket 2 and have contributed libraries.
My hope is that they add good parallelism support in Racket 7, and since it's based on Chez scheme all thread support should already be built in. Places suck.
Before you criticize me for this statement, please let me assure you that it's based on plenty of experience. I'm using Racket for almost all my programming since Racket 2 and have contributed libraries.
My hope is that they add good parallelism support in Racket 7, and since it's based on Chez scheme all thread support should already be built in. Places suck.
It does, in at least two different ways:
1. via `places` - basically spawning a new OS-level process and communicating with it. Has an advantage of being able to talk to processes on other machines.
2. via `futures` - OS-level threads, which insert synchronization points automatically where necessary. You have to structure your code in a particular way to make them not to sync, at which point you get a safe parallelism within a process. You get a future visualizer which helps in finding unsafe (syncing) memory accesses.
So the support for parallelism may be nowhere near Erlang or Pony, a bit worse than Java and a bit better than Python or Ruby.
1. via `places` - basically spawning a new OS-level process and communicating with it. Has an advantage of being able to talk to processes on other machines.
2. via `futures` - OS-level threads, which insert synchronization points automatically where necessary. You have to structure your code in a particular way to make them not to sync, at which point you get a safe parallelism within a process. You get a future visualizer which helps in finding unsafe (syncing) memory accesses.
So the support for parallelism may be nowhere near Erlang or Pony, a bit worse than Java and a bit better than Python or Ruby.
It definitely seems like it does https://docs.racket-lang.org/guide/parallelism.html
> Because Crystal is compiled, it is impossible to have a true REPL
First of all, the problem isn't that it's "compiled" - Python is "compiled" too, but still has one of the best REPLs available.
The problem is that the language is static. But static (and compiled) languages can have REPLs, for example C++ has an excellent REPL called Cling.
It's not impossible, just requires some more effort.
As for Crystal, it looks like a neat language. However, if it's going to try to compete with the fast compiled runtimes such as Java, it will need some parallelism support.
First of all, the problem isn't that it's "compiled" - Python is "compiled" too, but still has one of the best REPLs available.
The problem is that the language is static. But static (and compiled) languages can have REPLs, for example C++ has an excellent REPL called Cling.
It's not impossible, just requires some more effort.
As for Crystal, it looks like a neat language. However, if it's going to try to compete with the fast compiled runtimes such as Java, it will need some parallelism support.
No, it's not because it's compiled, it's because of the global type inference algorithm (which is very different from any existing type inference algorithm). Crystal used to have a repl but it was removed because it was buggy and difficult to maintain. Maybe crystal will get a repl in the future, but I wouldn't count on it because it's a huge effort.
Yeah, to clarify this should be possible, but I don't think it's going to happen for some time. There is a shard that creates a REPL, but it re-executes all previously executed commands each time you enter something new, which is super bad and dangerous. What we need is something that will compile just the line that was entered, and somehow "inject" that line into the existing program, and execute only that next line. Also there might always be a bit of lag between when you enter a line and when you get your response because the compiler is invoked. I'm not sure if that could be optimized to the point where it isn't noticable, but that would be super cool. I added on functionality for the Amber framework that allows you to execute one liners via `amber exec "code here"` as a replacement for `rails console` (amber is basically the rails of the crystal world).
Both scala and Java provides a REPL by compiling each lines + maintaining a context containing all local variables, functions, classes, etc
Check Swift Playground demos.
> First of all, the problem isn't that it's "compiled" - Python is "compiled" too, but still has one of the best REPLs available.
In a way it is. For a REPL you most likely need an interpreter for your language, like Cling is for C++.
In a way it is. For a REPL you most likely need an interpreter for your language, like Cling is for C++.
You don't need an interpreter for a REPL. Swift's REPL performs line-at-a-time compilation, running the program to that line and suspending it until you write and compile another line.
>You don't need an interpreter for a REPL. Swift's REPL performs line-at-a-time compilation
Isn't that another way to say "interpreter"?
Isn't that another way to say "interpreter"?
For all practical purposes, yes. In the strictest technical sense it's a form of JIT compilation, but that can be considered an optimization or implementation technique. If the user doesn't perceive a separate compilation step, it's effectively an interpreter and only a dedicated pedant would "well actually" that it's compilation.
Not really. An interpreter doesn't convert the language syntax tree into machine code, instead it just performs the operations. It sounds like Swift's REPL actually converts the lines into opcodes and runs those.
It can be argued that "compiled/interpreted" is not a property of programming languages, but of programming language implementations.
I second this.
> Recently Crystal shocked the world when it rose from 60th place to 32nd place in the Tiobe index in a mere month
Meteoric! Because we all know the the Tiobe index is incredibly accurate.
Would have been nice to see some code examples, rather than just handwavey bullets points that you could write about nearly every up-and-coming programming language.
Meteoric! Because we all know the the Tiobe index is incredibly accurate.
Would have been nice to see some code examples, rather than just handwavey bullets points that you could write about nearly every up-and-coming programming language.
> Because we all know the the Tiobe index is incredibly accurate.
More importantly, the TIOBE index is just a web noise indicator. A dedicated marketing guy focused on dumping the project's keywords throughout the WWW is all it takes to manipulate the ranking.
More importantly, the TIOBE index is just a web noise indicator. A dedicated marketing guy focused on dumping the project's keywords throughout the WWW is all it takes to manipulate the ranking.
Also, "meteoric rise" is kind of silly. That's not how meteors work.
A meteoric rise means a rise up into the sky.
Meteoros is Latin and means "to raise up", usually into the sky. Hence "meteorology" which is the study of the sky and "meteors" which are bright lights in the sky.
Meteoros is Latin and means "to raise up", usually into the sky. Hence "meteorology" which is the study of the sky and "meteors" which are bright lights in the sky.
"Well actually ..." but still no.
People don't say "meteorological rise" do they? They're not talking about gradual lifting, or about eyes. They're talking about a meteor. That's a real English word for a real physical thing, which by definition is approaching the earth - i.e. falling. Even if there's some etymological relationship to rising, the dominant relationship between the metaphor and the thing to which it refers in practically any listener's mind is all screwed up. Learn a better phrase, or even better make one up, instead of rationalizing one that's broken.
People don't say "meteorological rise" do they? They're not talking about gradual lifting, or about eyes. They're talking about a meteor. That's a real English word for a real physical thing, which by definition is approaching the earth - i.e. falling. Even if there's some etymological relationship to rising, the dominant relationship between the metaphor and the thing to which it refers in practically any listener's mind is all screwed up. Learn a better phrase, or even better make one up, instead of rationalizing one that's broken.
I believe "meteoric" mainly refers to the speed, not to the sky-reference.
TIL, thanks
Meteors fall, so they must also rise. Because physics?
On the TIOBE index homepage it seems to be currently (June 2018) listed in the 61st place.
hey guys OP here just a note on parallelism -- once windows support is done (very soon), parallelism is the #1 priority of the dev team. If you are looking for syntax examples check out the docs there are tons of examples here: https://crystal-lang.org/docs/syntax_and_semantics/
I think not having a parallelism first mindset will be the killer of the language. It's too important in today's world to not have that be an essential part of the language from the beginning. Rust and go for example we're 100% designed around parallelism.
I think you're getting confused by concurrency and parallelism. Crystal is concurrency-first and uses the same concurrency model as go. Expanding that to run fibers on multiple cores is just an implementation detail.
A difficult implementation detail, but it shouldn't affect the core of the language.
A difficult implementation detail, but it shouldn't affect the core of the language.
You posted a message that's essentially hand-waving. Discussing semantics means nothing if the language supports neither parallelism nor concurrency.
But crystal support concurrency, and it supports it very well. That's why it achieves about 100,000 HTTP requests per second per core, you just can't get that kind of performance on a single core without using evented IO.
>I think not having a parallelism first mindset will be the killer of the language.
"Parallelism first"? That's a contentless statement that borders on marketing speak...
"Parallelism first"? That's a contentless statement that borders on marketing speak...
I'm reading through the list and it looks like D has all these, except the ninth one ("meteoric rise in popularity"). However, it does have binary compatibility with C and the binaries are much leaner.
> Because Crystal is compiled, it is impossible to have a true REPL
This has nothing to do with being a "compiled" language, nor with being a statically types language for that matter.
If languages like Haskell and Idris can have a repl, very basic languages like Crystal certainly can. I can imagine it not being their top priority though.
This has nothing to do with being a "compiled" language, nor with being a statically types language for that matter.
If languages like Haskell and Idris can have a repl, very basic languages like Crystal certainly can. I can imagine it not being their top priority though.
This is what I appreciate about HN: reading that I was pretty convinced and had no idea what Crystal's shortcomings might be.
Then I read the comments and the largest single complaint is lack of parallelism, by which people seem to mean being able to utilize all the cores of the processor.
I've been programming for a while, but haven't run into this issue.
Can someone explain who uses parallelism, for what, and in what context?
Then I read the comments and the largest single complaint is lack of parallelism, by which people seem to mean being able to utilize all the cores of the processor.
I've been programming for a while, but haven't run into this issue.
Can someone explain who uses parallelism, for what, and in what context?
Parallelism is large part of the answer to question of "how do we make this thing as fast as hardware allows" (most people will add "easily" to that). Or "as fast as our users demand" if it makes it easier to understand. The use for that comes with scale, there is a one caveat though: it is often impossible to add this as an afterthought to your program and to programming language. If you choose programming language (or architecture) that does it poorly, you are digging yourself a hole you won't be able to get out of. This is main reasons why it is important even for cases when you don't need it right now.
Thank you for taking the time to answer the question.
In principle, I know how it ought to work. But when would I need to use it?
For instance, if someone is making a web api, and you're pulling data from a database, should one think about it?
If one is doing some data analysis, should one think about it?
I always thought that it is a "low level" program that takes care of it. So tensorflow might worry about it, but I wouldn't if I use tensorflow. Or perhaps mapreduce worries about it, or the ORM worries about it.
But when should I worry about it?
In principle, I know how it ought to work. But when would I need to use it?
For instance, if someone is making a web api, and you're pulling data from a database, should one think about it?
If one is doing some data analysis, should one think about it?
I always thought that it is a "low level" program that takes care of it. So tensorflow might worry about it, but I wouldn't if I use tensorflow. Or perhaps mapreduce worries about it, or the ORM worries about it.
But when should I worry about it?
If all you ever deal with are what are known as "embarrassingly parallel" problems, where you don't need to do any synchronization or intermediate communication between cores, then true parallelism isn't really needed and you can do whatever you were trying to do with simple process forking or something of that nature. Problems that are not embarrassingly parallel often require frequent and complicated communication and synchronization between cores. For this you need to use a real multi-threading library, or do some complicated message passing on a per-process level. If you need hundreds of workers, it is better to use something more light-weight than a thread, as Go does. Once crystal's parallelism support is added, it will have these light-weight "go-style" threads, so this complaint will disappear.
I don't know your experience so I'll do my best to draw on examples _I suspect_ you might be familiar with. I'll try to keep this simplified but deep enough. I'm taking a shot in the dark and hoping you're more familiar with the one of the python/node/ruby world.
You should worry about it when you want to do something faster than you are already or want to do more things at once. Maybe someone's framework/service has done it for you maybe not.
For a web api, the workload _usually_ looks like: {database call} -> {do some work} -> {maybe more calls} -> {more local work} -> {done}.
In the python/node/ruby world where you have mostly single process workers. You get some parallelism by doing work locally while the database call is made asynchronously. If you want to handle more requests on a host you'll just run more processes and load balance across the front (nginx, gunicorn, etc). In languages with threads you usually do that INSIDE your program. A single process load balances across worker threads internally.
At first blush, multiple threads are pretty similar to just running multiple programs. It gets more complicated when you start talking about how you want to communicate between them or share resources across them.
For data analysis you often have algorithms that are embarrassingly parallel[0] chained together. You use threads the same way you'd call three friends to help you move. You all pack boxes at the same time without really needing to coordinate and then you start loading the truck where you have to talk more. One might wait in the truck and organize while 2 or 3 others might be shuttling boxes outside. Maybe everyone sits around while the stairs are blocked by one person moving a shelf. It just depends. Having 4 people helps sometimes or is disruptive in others.
[0]: https://en.wikipedia.org/wiki/Embarrassingly_parallel
P.S. - To address your examples: tensorflow, map-reduce, orms. etc.
It's all still about parallelism but not necessarily about threads.
Parallelism is just doing more things at the same time. Threads are (mostly) about CPU parallelism, but many of those use other resources to get parallelism. Tensorflow might use the GPU, which has many threads internally (simplification), to do one thing while the CPU does another. Mapreduce will use a lot of different machines across a cluster, but maybe only a thread or two on each, to archive parallelism (and really getting disk IO parallelism). And ORM will call the database and let it do work while your CPU else instead of wait.
You should worry about it when you want to do something faster than you are already or want to do more things at once. Maybe someone's framework/service has done it for you maybe not.
For a web api, the workload _usually_ looks like: {database call} -> {do some work} -> {maybe more calls} -> {more local work} -> {done}.
In the python/node/ruby world where you have mostly single process workers. You get some parallelism by doing work locally while the database call is made asynchronously. If you want to handle more requests on a host you'll just run more processes and load balance across the front (nginx, gunicorn, etc). In languages with threads you usually do that INSIDE your program. A single process load balances across worker threads internally.
At first blush, multiple threads are pretty similar to just running multiple programs. It gets more complicated when you start talking about how you want to communicate between them or share resources across them.
For data analysis you often have algorithms that are embarrassingly parallel[0] chained together. You use threads the same way you'd call three friends to help you move. You all pack boxes at the same time without really needing to coordinate and then you start loading the truck where you have to talk more. One might wait in the truck and organize while 2 or 3 others might be shuttling boxes outside. Maybe everyone sits around while the stairs are blocked by one person moving a shelf. It just depends. Having 4 people helps sometimes or is disruptive in others.
[0]: https://en.wikipedia.org/wiki/Embarrassingly_parallel
P.S. - To address your examples: tensorflow, map-reduce, orms. etc.
It's all still about parallelism but not necessarily about threads.
Parallelism is just doing more things at the same time. Threads are (mostly) about CPU parallelism, but many of those use other resources to get parallelism. Tensorflow might use the GPU, which has many threads internally (simplification), to do one thing while the CPU does another. Mapreduce will use a lot of different machines across a cluster, but maybe only a thread or two on each, to archive parallelism (and really getting disk IO parallelism). And ORM will call the database and let it do work while your CPU else instead of wait.
Great answer. You should post that on a blog if you have one.
This was the best part if you want to expand on it in a longer post: "You should worry about it when you want to do something faster than you are already or want to do more things at once. Maybe someone's framework/service done it for you maybe not.
For a web api, the workload _usually_ looks like: {database call} -> {do some work} -> {maybe more calls} -> {more local work} -> {done}.
In the python/node/ruby world where you have mostly single process workers. You get some parallelism by doing work locally while the database call is made asynchronously. If you want to handle more requests on a host you'll just run more processes and load balance across the front (nginx, gunicorn, etc). In languages with threads you usually do that INSIDE your program. A single process load balances across worker threads internally.
At first blush, multiple threads are pretty similar to just running multiple programs. It gets more complicated when you start talking about how you want to communicate between them or share resources across them."
This was the best part if you want to expand on it in a longer post: "You should worry about it when you want to do something faster than you are already or want to do more things at once. Maybe someone's framework/service done it for you maybe not.
For a web api, the workload _usually_ looks like: {database call} -> {do some work} -> {maybe more calls} -> {more local work} -> {done}.
In the python/node/ruby world where you have mostly single process workers. You get some parallelism by doing work locally while the database call is made asynchronously. If you want to handle more requests on a host you'll just run more processes and load balance across the front (nginx, gunicorn, etc). In languages with threads you usually do that INSIDE your program. A single process load balances across worker threads internally.
At first blush, multiple threads are pretty similar to just running multiple programs. It gets more complicated when you start talking about how you want to communicate between them or share resources across them."
Thanks, I'm glad it was helpful! I don't have a blog yet, but I started commenting a month back to build up some writing chops so I'm especially happy to hear it.
Crystal 0.25.0 has just been released, you can check out the new features here: https://crystal-lang.org/2018/06/15/crystal-0.25.0-released....
Looks cool, in a similar vein as Zig or NIM it seems?
A few questions:
Out of curiosity, is there a company behind Crystal? or is it a BDFL situation? The press on Crystal has been pretty significant lately which is great. I ask mostly out of historical interest. IIRC Rust/Go seem to had company backing while gaining traction but Python/Ruby were more organic BDFL led projects. Naturally C/Java/C# were literal company products.
While the Ruby-like syntax is not to my taste (prefer Python) it got me thinking about what are the big syntax families? Off the top of my head I count: Lisp, C, Python, Ruby, Prolog, ML, APL? Preemptively apologizing for only knowing the popularizers over the originals.
A few questions:
Out of curiosity, is there a company behind Crystal? or is it a BDFL situation? The press on Crystal has been pretty significant lately which is great. I ask mostly out of historical interest. IIRC Rust/Go seem to had company backing while gaining traction but Python/Ruby were more organic BDFL led projects. Naturally C/Java/C# were literal company products.
While the Ruby-like syntax is not to my taste (prefer Python) it got me thinking about what are the big syntax families? Off the top of my head I count: Lisp, C, Python, Ruby, Prolog, ML, APL? Preemptively apologizing for only knowing the popularizers over the originals.
I think some of the comments here are unfair to Crystal and the careful thought (and lots of work!) that has been put into it. Projects like this should not be so easily dismissed as "also-rans".
I read through the Crystal language docs (meaning just the syntax etc) and as a seasoned C++ and Python developer who is _constantly_ looking for something with better performance (than Python) yet much cleaner (than C++), I think Crystal has a lot going for it so far.
Any "great" language should be able to take a thought in a developer's head and easily allow 1) the concise expression of that thought, and 2) efficient evaluation of that thought. I mean, those things we probably would all agree on.
It saddens me to say this, but C++ is falling over as a result of it's own weight. It's become a language for experts. Perhaps more than any popular language, it can take a simple idea in a developer's head and turn it into pages of code. It's actually quite embarrassing. I won't go into that further; judge for yourself (and sorry if that comment offends anyone -- I love C++ and use it every day). But you just can't beat the speed... Well-crafted C++ _should_ exceed the speed of even C (Why? Because templates...). As an aside, it's disingenuous to put Java in the same speed category as C/C++... The "fast" Java programs out there are basically C with Java wrappers (ducks thrown tomatoes). And just like C++, Java is very noisy (but for different reasons).
Python as we all know sort of takes the opposite approach, with dynamic typing a design-as-you-go mentality. And boy what a success it has been, with a flourishing package ecosystem. There's lots of good things to say about Python, but it's f*cking slow as hell (Cython is a hack, Numba shows promise, but PyPy isn't much faster... I was excited about Pyston but don't know where that went). It's not the fault of Python that it's slow -- it's the price of such a wonderfully dynamic language.
So enter things like Crystal. And trust me -- it's definitely early days with this language. But I like the fact that the designers really seem to care about the things that (to most of us I think) matter.... Taking an idea in our brain and putting it (simply) down in code, and then having that code run quickly. Yay!
In this day and age where we are swamped with hype from all of these new languages, let's give praise where it's warranted -- to the people are out there that are trying to refine decades worth of thought and finally "get it right".
My hat is off to those people out there that are forging ahead with these types of projects. Don't mind the criticism -- keep it up and great job.
I read through the Crystal language docs (meaning just the syntax etc) and as a seasoned C++ and Python developer who is _constantly_ looking for something with better performance (than Python) yet much cleaner (than C++), I think Crystal has a lot going for it so far.
Any "great" language should be able to take a thought in a developer's head and easily allow 1) the concise expression of that thought, and 2) efficient evaluation of that thought. I mean, those things we probably would all agree on.
It saddens me to say this, but C++ is falling over as a result of it's own weight. It's become a language for experts. Perhaps more than any popular language, it can take a simple idea in a developer's head and turn it into pages of code. It's actually quite embarrassing. I won't go into that further; judge for yourself (and sorry if that comment offends anyone -- I love C++ and use it every day). But you just can't beat the speed... Well-crafted C++ _should_ exceed the speed of even C (Why? Because templates...). As an aside, it's disingenuous to put Java in the same speed category as C/C++... The "fast" Java programs out there are basically C with Java wrappers (ducks thrown tomatoes). And just like C++, Java is very noisy (but for different reasons).
Python as we all know sort of takes the opposite approach, with dynamic typing a design-as-you-go mentality. And boy what a success it has been, with a flourishing package ecosystem. There's lots of good things to say about Python, but it's f*cking slow as hell (Cython is a hack, Numba shows promise, but PyPy isn't much faster... I was excited about Pyston but don't know where that went). It's not the fault of Python that it's slow -- it's the price of such a wonderfully dynamic language.
So enter things like Crystal. And trust me -- it's definitely early days with this language. But I like the fact that the designers really seem to care about the things that (to most of us I think) matter.... Taking an idea in our brain and putting it (simply) down in code, and then having that code run quickly. Yay!
In this day and age where we are swamped with hype from all of these new languages, let's give praise where it's warranted -- to the people are out there that are trying to refine decades worth of thought and finally "get it right".
My hat is off to those people out there that are forging ahead with these types of projects. Don't mind the criticism -- keep it up and great job.
I'm similar to you. An every-day / almost every day C++ user. I've been using Clojure for a personal project and I really enjoy the much-reduced overhead imposed by the language. Unfortunately it runs on the JVM so that's not something I'm super stoked about. But this language seems interesting. The interoperability with C libs gives it a head start. The syntax seems much nicer than Rust. Too bad about the parallelism but frankly you can get by with a single core for a lot of things, I'm sure it's on the roadmap anyhow.
Lisps are all very, very cool -- you basically get to build your own world. And the macros -- so powerful. But the parens! Not the number of them, as that is about the same as other langs, but the placement of them:
Sample factorial function:
(define fac (lambda (n) (if (= n 0) 1 (* n (fac (- n 1)))))) ; <<--- look at those parens!
And with Clojure, you don't have proper tail recursion, so you'll have to add some Clojure-only thing in there to prevent the above code from blowing up for large numbers.
There's really no getting around it -- Lisp, for many people, is just hard to parse. Consider:
def fac(n): n < 2 ? n : n * fac(n - 1)
Yay! No noise. But I agree, there are some cool things about Clojure.
Sample factorial function:
(define fac (lambda (n) (if (= n 0) 1 (* n (fac (- n 1)))))) ; <<--- look at those parens!
And with Clojure, you don't have proper tail recursion, so you'll have to add some Clojure-only thing in there to prevent the above code from blowing up for large numbers.
There's really no getting around it -- Lisp, for many people, is just hard to parse. Consider:
def fac(n): n < 2 ? n : n * fac(n - 1)
Yay! No noise. But I agree, there are some cool things about Clojure.
How do we know that
One level of parentheses comes from using a dialect which uses a variable definition and lambda to define a function, instead of having function-defining syntax like defun.
You've loaded the readability dice by writing it on one line. To help with the readability issues in both languages, we can use multiple lines and indentation:
def fac(n): n < 2 ? n : n * fac(n - 1)
def fac(n): n < 2 ? n : n * fac(n - 1)
+ more stuff
One level of parentheses comes from using a dialect which uses a variable definition and lambda to define a function, instead of having function-defining syntax like defun.
n < 2 ? n : n * fac(n - 1)
You've loaded the readability dice by writing it on one line. To help with the readability issues in both languages, we can use multiple lines and indentation:
def fac(n):
n < 2
? n
: n * fac(n - 1)
(defun fac (n)
(if (< n 2)
n
(* n (fac (- n 1)))))yes, lisp is a fossil of a time before precedence was invented.
what I dream of is a Lisp with an additional mechanism to define precedence of functions/operators. And infix operators. Almost like Haskell.
what I dream of is a Lisp with an additional mechanism to define precedence of functions/operators. And infix operators. Almost like Haskell.
> mechanism to define precedence
Read syntax:
https://en.wikipedia.org/wiki/CGOL [1973!]
https://www.cliki.net/infix
Macros:
https://stackoverflow.com/questions/25434538/define-function...
Read syntax:
https://en.wikipedia.org/wiki/CGOL [1973!]
https://www.cliki.net/infix
Macros:
https://stackoverflow.com/questions/25434538/define-function...
I never seem to be able to get into all of these new languages, mostly because I do not see a clear use case for it. The last two languages I learnt were Python, because we have our backend written in it and Swift for a personal project because it is what one writes iOS apps in today.
I fail to imagine a kind of application that would warrant to learn a new general purpose language on top of what I already know. Well, except massive parallelism for which I would probably reach for Erlang or Go.
I fail to imagine a kind of application that would warrant to learn a new general purpose language on top of what I already know. Well, except massive parallelism for which I would probably reach for Erlang or Go.
At BlockVue we are using crystal in production to solve some very computationally expensive constraints programming problems that we would otherwise likely have to tackle in C/C++.
Is this something which is inherently better handled by Crystal? For example does it have good libraries for contraint solving? What was your main motivation to choose Crystal?
The next big programming language will compile to WebAssembly, crystal doesn't (yet).
Yeah I'm really crossing my fingers for this one as well. Imagine writing all your server side code in crystal, as well as your front end. Droool..
> Because Crystal is compiled, it is impossible to have a true REPL
Impossible, or just hard?
Impossible, or just hard?
I had a quick search, REPLs exist for C, C++, D, Go, and Rust (my shortlist of statically typed, compiled languages).
I would question what is meant by a 'true REPL'. It's certainly seems possible to build a REPL of a statically typed compiled language.
I would question what is meant by a 'true REPL'. It's certainly seems possible to build a REPL of a statically typed compiled language.
I don't know about the others (reportedly Cling for C++ is good) but the Rust repl doesn't work properly and is basically just a frontend for incrementally writing a Rust file.
Not only that but it’s about a year out of date at this point.
I’m hoping someone eventually makes a new one, possibly based on MIRI.
I’m hoping someone eventually makes a new one, possibly based on MIRI.
This is also the current state of the crystal repl shard
Haskell static, compiled, and has a REPL, yes? Is the Haskell REPL not useful?
You'd need an interpreted runtime for the language for that to work. D has a 'scripting' runtime but I'm not sure if anyone's done a REPL or what the limitations would be.
/smug-lisp-weenie-mode
I didn't realise that SBCL is interpreted.
I didn't realise that SBCL is interpreted.
I'm not sure how SBCL is implemented, but I've written my own Lisp. It's incrementally compiled to byte code - the compiler is called when defun or defmacro is evaluated - but the REPL still relies on the interpreter, the same interpreter which does everything until the compiler compiles itself during a clean build.
So compiled code can have a REPL, but a REPL needs an interpreter as well.
So compiled code can have a REPL, but a REPL needs an interpreter as well.
SBCL compiles always. Code from the REPL is also compiled.
OTOH LispWorks uses an interpreter for the REPL interaction.
Yes, an Interpreter offers some more interactive features - but usually lacks the compile-time warnings/errors.
OTOH LispWorks uses an interpreter for the REPL interaction.
Yes, an Interpreter offers some more interactive features - but usually lacks the compile-time warnings/errors.
The main difference between compiled and static languages is that compiled languages go through a phase of "checking your work", where it compares parts of your program to other parts before running it. For example, if you misspelled a name, the compiler will look everywhere in your program for that name before you run it, and tell you it can't find the name. That's why we like compiled languages. By contrast, dynamic language handle this by looking up the name right before the code in question is executed. If the name's not there at that time, an error is thrown.
Now think about this in the context of a REPL, which again stands for Read, Evaluate, Print, Loop. If you write code and evaluate it right away, this whole business about checking your work won't work right. You would basically have to recompile your entire program on every REPL eval, which defeats the whole purpose of a REPL. There are lots of other tricks to getting this to work, as the article hints at, but they're all very unsatisfactory in delivering the freedom of a dynamic language.
Now think about this in the context of a REPL, which again stands for Read, Evaluate, Print, Loop. If you write code and evaluate it right away, this whole business about checking your work won't work right. You would basically have to recompile your entire program on every REPL eval, which defeats the whole purpose of a REPL. There are lots of other tricks to getting this to work, as the article hints at, but they're all very unsatisfactory in delivering the freedom of a dynamic language.
Really?
SBCL:
SBCL:
* (defun foo (a) (+ a 42))
FOO
* (defun bar (b) (foo 10 20))
; in: DEFUN BAR
; (FOO 10 20)
;
; caught STYLE-WARNING:
; The function was called with two arguments, but wants exactly one.
; (SB-INT:NAMED-LAMBDA BAR
; (B)
; (BLOCK BAR (FOO 10 20)))
;
; caught STYLE-WARNING:
; The variable B is defined but never used.
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions
BARThat's not the point. You can run static languages in a REPL. Some easier than others. You've chosen a particularly easy static language to write REPLs for as a counterpoint. LISPs and other homoiconic languages are especially well suited for interpreters because they're especially well designed languages.
Languages like C which require header files and a pre-processor are much harder to write REPLs for. You can try to do it and you can build a thing that works kind of like a REPL (look I type code and press enter and it evaluates, yay), but the experience always falls short of using a dynamic language in a REPL. It's just not even close, and that's just due to the simple fact you're trying to use something in a way it's not intended. Compiled languages simply are not designed with a tight development loop as a major design goal.
Languages like C which require header files and a pre-processor are much harder to write REPLs for. You can try to do it and you can build a thing that works kind of like a REPL (look I type code and press enter and it evaluates, yay), but the experience always falls short of using a dynamic language in a REPL. It's just not even close, and that's just due to the simple fact you're trying to use something in a way it's not intended. Compiled languages simply are not designed with a tight development loop as a major design goal.
SBCL is compiled, dynamic and supports REPL based programming.
Like I said, Lisps are well suited for REPLs.
This explains it a lot better than I do in the article.
And yet I got downvoted.
Because it's not correct.
The correct response to a wrong post is to correct it, not to drive by downvote. Explain where I'm wrong and then we have the basis for a discussion on the merits. Otherwise you're not adding anything.
There are plenty of corrections in the siblings.
But I'm a bit vary as for the casual observer the Crystal project github (https://github.com/crystal-lang/crystal) seems brim with unsolved issues (over 500 issues!) and stale PR-s (over 105!). I'm not sure if this is a sign of poor project managment, development stall or poor community engagement but still it leaves a bad first impression. Contrast this to elixir project page (https://github.com/elixir-lang/elixir).
Maybe someone who is actively involved in the development of Crystal could shine a light?