Typing lists and tuples in Elixir(elixir-lang.org)

231 pointsby idmitrievsky2 anni fa52 comments

https://elixir-lang.org/blog/2024/08/28/typing-lists-and-tuples/

52 comments

ku1ik2 anni fa

I really respect Elixir core team’s approach to adding gradual typing to the language. They don’t rush it. They didn’t put too much focus on syntax so far (I’d argue the syntax in many cases is less important than foundations) and instead they focused on soundness of the system. With each new Elixir version the compiler is getting smarter, catching more bugs. Not hugely smarter, but smarter enough that I feel safer. Looking forward to Elixir 1.18!

jonnycat2 anni fa

I'm not sure... I'm a huge Elixir fan and I trust José to build a great solution, but I've found the rollout to be a bit confusing. There was the announcement that "Elixir is now a gradually typed language" prior to 1.17 - but it seems that most of the changes were behind the scenes, and 1.17 largely didn't expose user-facing type errors or warnings.

Again, I definitely trust them to get it right in the long term, but in the meantime, the progress has been a bit confusing to me.

josevalim2 anni fa

Thanks for vote of confidence!

We need to type every data type and every function, so the type system will be rolled out over a long period of time.

The 1.17 release meant that we now have a gradual type system, which runs in every code being compiled, but it only supports a handful of types (including the dynamic one). The full list of supported types and examples of typing violations it can now detect is on the announcement: https://elixir-lang.org/blog/2024/06/12/elixir-v1-17-0-relea...

There is no support for type annotations, that comes in a later stage. The overall stages have been described in an earlier article (and I believe also in the paper): https://elixir-lang.org/blog/2023/06/22/type-system-updates-...

klibertp2 anni fa

> but it seems that most of the changes were behind the scenes, and 1.17 largely didn't expose user-facing type errors or warnings.

That's how it normally goes with gradual type systems for existing languages, I think. The first step seems to be almost always adding a type checker that doesn't do anything in particular other than handling untyped code. Since being able to handle untyped code makes a type system gradual, announcing Elixir as "gradually typed" when this milestone is reached seems justified. After that, you're free to improve the type system and type checker(s), improve type inference, add specialized syntax, improve typed/untyped interactions, cover more language patterns, and so on. MyPy for Python also started without support for many things that were added later (and it's still being actively developed ten years later).

sodapopcan2 anni fa

The wording was a little odd, but there are certainly user-facing errors in 1.17, namely:

- Map keys (called with '.') are checked at compile time.

- Using comparison operators with different types causes a warning.

I may be forgetting something.

munchler2 anni fa

F# has both a `head` and `tryHead` function to handle lists that may or may not be empty. In general, `tryFoo` is a good pattern for naming functions that might fail.

Having a separate NonEmptyList type might seem like a good idea in theory, but in my experience, it leads to code that is significantly more complicated.

bmitc2 anni fa

Just to clarify for the crowd though. In F#, `List.head` throws an exception when it fails whereas `List.tryHead` returns an `option`, which returns `None` when it fails instead of an exception.

A general confusion of mine in Elixir is generally how libraries and functions treat errors. There's the common idiot of returning either `{:ok, ____}` or `{:error, ____}`, but what can be inside the error tuple is not always clear. The other thing is that sometimes a function can both throw an exception and also return a success tuple. Such cases are confusing to handle, and there's a large gap between handling cases like that and the philosophy of "let it crash", which I think is preached a little looser than it should actually be practiced.

I do like F#'s way of disambiguating the two situations. The only issue I have in F#, which actually exists in every language that I know of that has exceptions, is that there is no way to know, up front and clearly, what exceptions can be thrown by a given function. This is particularly frustrating in F#, which has fantastic pattern matching for exceptions. I wish there was exhaustive pattern matching in F# for exception handling, such that it would warn you that you have an unhandled exception in a try/with expression (https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...) but of course would allow for wildcard patterns.

sodapopcan2 anni fa

> There's the common idiot of returning either `{:ok, ____}` or `{:error, ____}`, but what can be inside the error tuple is not always clear.

That's the result of Elixir being dynamic and not having built-in monads. You always have to check the docs/code.

> The other thing is that sometimes a function can both throw an exception and also return a success tuple.

I would say this is just poor design (though maybe someone could point me to an exemplary use of this?). As I see it (which is from how I've read it described and how most good lib do it):

  - If function can error and the caller can do something about it, return :ok/:error.
  - If the caller can't do anything about the error, raise.
  - If the function can't fail, return a raw value.

bmitc2 anni fa

It's true that it's a poor library implementation, but it's also a fairly natural consequence of a language with exceptions plus some other way of handling errors.

Even in a language like F#, just becauase a function returns an `option` type doesn't mean it won't also throw an exception sometimes. However, I don't necessarily think pure functional languages solve this either. If a Haskell function returns an option, you have little idea as to where the error originated. There are error types such as result types, which F# has as well, but then that's basically back to exceptions, perhaps even as a more limited form.

I'm curious if there's a language that's really nailed error handling. Erlang/Elixir have their supervisors and functional languages have pattern matching on option types, result types, and exceptions, but surely there's a way to improve on that.

IshKebab2 anni fa

Java has checked exceptions and they are used, e.g. in Android. But it seems to be the exception (heh) rather than the rule.

IMO the problem is proper exception handling with checked exceptions and wrapping each function (or at least small blocks) in try catch is just so insanely verbose that even though it is possible to get error handling as good as something like Rust, nobody actually does it in practice.

truculent2 anni fa

How does it make it more complicated?

In my view, you’re moving the potential for failure to a different place (the constructor), rather than changing some fundamental property or introducing new complexity.

Is it handling the construction of these types you find complicated? And is it simply not worth the guarantees?

greener_grass2 anni fa

The intuitive definition of NonEmptyList is:

    type NonEmptyList<'t> = 't * 't list

But this cannot be passed to any function that expects a List<'t>. This is odd though, since intuitively, all non-empty lists are lists.

See Rich Hickey's "Maybe Not" talk

OOP solution is to use inheritance. Typical ML solution is to use type-classes.

F# sits in an awkward middle-ground where neither is a perfect fit.

I believe that dependently-typed languages solve this more elegantly.

There's also the syntactic inconvenience of wrapping at construction, where in theory the compiler could figure it out for you.

For example:

    let xs = 1 :: 2 :: 3 :: []

Here xs is non-empty, but we must tell the compiler:

    let xs = 1, 2 :: 3 :: []

TypeScript does a better job here (although a great cost!)

What you end up with is massive code duplication or lots of extra function calls:

   xs
   |> NonEmptyList.toList
   |> List.map (fun x -> x + 1)
   |> NonEmptyList.unsafeFromList

(I say this all as someone who really likes F#)

josevalim2 anni fa

We (Elixir) would rather define lists on top of non empty lists:

    list(a) = empty_list() or non_empty_list(a)

So you should pass non-empty lists everywhere a list is expected. But you can’t pass a list where a non-empty one is expected.

But overall, you are right: our concern is exactly all of the extra function calls that may now suddenly become necessary (and the tension mentioned in the article). We will review our design decisions as we keep on rolling out the type system!

greener_grass2 anni fa

Does this break down if we want some other types?

e.g.

    list(a) = empty_list() or singleton_list(a) or two_or_more_list(a)

yawboakye2 anni fa

for when one expects a list to be non-empty, i think there’s strong argument in favor of an enforcement from the type checker, given that the prove will very likely be necessary. if not in the application code then in the tests.

jerf2 anni fa

"OOP solution is to use inheritance. Typical ML solution is to use type-classes."

Yes, type classes can "work" to help a NonEmptyList degenerate to a normal List of some sort, if the function accepting the list accepts the type class instead of a hard-coded List. Unfortunately, at least for this exact task, taking hard-coded types is pretty common. I've sometimes wondered about the utility of a language that provided all of its most atomic types solely as typeclasses within its standard library, so that calling for a "List a" or "[a]" automatically was turned into the relevant type class.

Inheritance doesn't actually work here. I assume you mean inheriting a NonEmptyList from some sort of List, from the perspective of a user facing a language that has a standard List and they want to create a NonEmptyList that things taking List will accept. Unfortunately, that is a flagrant violation of the Liskov Substitution Principle and will create architecturally-fragile code.

Compilers can't enforce the LSP (with anything short of the dependently typed code you mention), so you can bash out a subclass that will throw an exception if you try to take the last element out of a NonEmptyList or violate the rules some other way, and if you pass your new NonEmptyList to something that happens to not do anything broken, you may get away with it, but by the standards of OO theory you're definitely "getting away" with something, not solving the problem.

I haven't studied this extensively beyond just thinking here for a moment, but I don't think you can LSP-legally go the other way either. A subclassed List can't revoke a parent's NonEmptyList property that the list is guaranteed to not be empty. Again, you can bash the methods into place to make it work, but as this is a very basic standard library sort of thing for a language it really needs to be right.

Edit: Yes, it's certainly illegal. You can take a List inherited from the NonEmptyList, have it be empty, but you have to be able to pass it to something accepting a NonEmptyList, but it will then be empty. So you can't LSP-legally inherit either way.

(This is one of the "deep reasons" why inheritance is actually not a terribly useful architectural tool. It technically breaks really, really easily... like, probably most non-trivial uses of inheritance in most code bases is actually wrong somehow, even if never happens to outright crash. We tend to just code past the problem and not think about it too hard.)

greener_grass2 anni fa

> You can take a List inherited from the NonEmptyList

Shouldn't it go the other way?

All NonEmptyLists are Lists, but not all Lists are NonEmptyList.

So NonEmptyList inherits from List

jerf2 anni fa

Neither direction works. More directly (since I was working it out as I typed above):

A NonEmptyList promises that its .Head method will always produce a value. An inherited List can not maintain that property, it must add either an error return or a possible exception (which is the same thing from this point of view), and so violates the LSP.

A List promises that if it has an element, you can remove it and have another List, whether by mutation or returning a new List. A NonEmptyList breaks that promise. If that sounds like a "so what", bear in mind that "removing an element" includes things like a "Filter" method, or a "split" method, or any of several other such methods beyond just iteration that a List is likely to have that a NonEmptyList is going to need a different type signature and/or exception profile to implement properly.

You could define a bare-bones superclass for both of them that allows indexing, iteration, appending, and a length method, without much else, and that does work. However, if you start trying to get very granular with that, across more data structures, you'll start to need multiple inheritance and that becomes a mess really quickly. There's a reason that, for instance, the C++ STL does not go the "inheritance hierarchy" route for this stuff.

Like I said, inheritance done properly is really restrictive. We often do a lot of sweeping under the rug without even realizing it, and that "works" but it still eats away at the architecture, all the more so if the people involved don't even realize what they are doing.

Volundr2 anni fa

> A List promises that if it has an element, you can remove it and have another List, whether by mutation or returning a new List. A NonEmptyList breaks that promise.

I don't follow. Remove the head from a NonEmptyList and the tail will be a List. It might not be a NonEmptyList, but that's not the contract of List.

greener_grass2 anni fa

NonEmptyList<'t> has a method Uncons that returns 't * List<'t>

List<'t> has a function TryUncons that gives Option<'t * List<'t>>

List<'t> does not have a method Uncons.

We can define TryUncons for NonEmptyList<'t> in terms of Uncons, specifically:

    this.TryUncons() = 
      this.Uncons() 
      |> Some

jerf2 anni fa

I didn't spell it out adequately, only implied it, but I am talking about a fully loaded List, not one that just barely works, e.g., it has filter, it has all the other things you'd expect from a List. I did discuss the "just barely works" case at the end.

It would be an odd "object oriented language" list that lacked such things, e.g., https://docs.oracle.com/javase/8/docs/api/java/util/List.htm... . We're in OO land here, not FP land.

ryangs2 anni fa

But the nonempty list never has an element, so we don't need to worry about the type mutation of removing an element from it. Filter just returns a nonempty list.

jerf2 anni fa

"But the nonempty list never has an element, so we don't need to worry about the type mutation of removing an element from it."

NonEmpty always has an element.

derriz2 anni fa

Why not in the other direction? If NonEmptyList inherits from List, then head (and tail) would be methods of NonEmptyList but not of List.

sdeframond2 anni fa

> A List promises that if it has an element, you can remove it and have another [..]. A NonEmptyList breaks that promise.

No. Removing an element from a NonEmptyList returns a List. LSP is respected when NonEmptyList is a List.

munchler2 anni fa

Yes, this is exactly what I was getting at. One ends up needing an explicit `toList` helper function that converts a non-empty list into a plain list.

Pattern matching on a non-empty list is also inelegant, because it is implemented as a tuple, which creates a leaky abstraction.

greener_grass2 anni fa

F# actually does have a good solution for pattern matching here, which is Active Patterns.

aloisdg2 anni fa

As a fellow F# dev currently learning Elixir, this kind of thing is a burden

yellowapple2 anni fa

Looks nice; I dig the $-prefixed type annotations. I also like that the types boil down to familiar conventions like atoms / atom-tagged tuples under the hood; that's exactly how I'd expect an Erlang/Elixir type system to work, and is exactly how I already tend to use tagged tuples.

> If we get rid of this limitations, we could define head as follows:

    $ non_empty_list(a) -> a
    def head([head | _]), do: head

In this case, I ain't sure what the typespec is really contributing here:

- We can already infer the input type, thanks to Erlang/Elixir baking pattern-matching into function signatures

- We can already infer the output type, because there's only one possible output and it's coming directly from the input

This is exactly the problem I hit the last time I tried to chase down the "typespec ALL the things!" path with Dialyzer: the typespecs are just restating what's already obvious from looking at the function, making them redundant at best. Yeah, having a summary of the input and output types is valuable for more complex functions, but it's already a common Erlang (and Elixir, by descent from Erlang and Ruby) best practice to break down complex functions into smaller, simpler units, at which point the typespecs lose value again.

I'm probably missing something here, though - and maybe a more complicated example would better illustrate the value these typespecs add.

btbuildem2 anni fa

I've always viewed Erlang (and by extension, Elixir) as safe from these types of improvements. Maybe I don't have enough experience in these languages, but having guards, and the different approach to conditionals seemed to do away with most of the pitfalls usually "safeguarded from" by the caution tape and excessive road signage of type systems.

I'm curious to learn more, but I can't shake a feeling of vague trepidation here.

rkangel2 anni fa

Erlang and Elixir have long had "dialyzer" as a type checker. The problem is that it had some severe limitations in approach (and implementation) and so the cost/benefit ratio wasn't great.

This article really speaks to how they're thinking about the costs of the type system, so that you mostly get benefit, and that's great.

__jonas2 anni fa

> having guards, and the different approach to conditionals seemed to do away with most of the pitfalls usually "safeguarded from" by the caution tape and excessive road signage of type systems

It doesn’t seem that way to me at all. The main pitfall static typing guards me from is runtime errors that can be easily avoided, guards don’t really help there.

If I do

  def double(num) when          is_number(num)  do
    num * 2
  end

I could have some call double(“4”) in my code somewhere, and if that’s reached it would throw a FunctionClauseError and crash the process. I don’t want to be able to do that, I want the compiler to scream at me when I write double(“4”) and not let me do it, the guard is doing nothing of the sort.

yellowapple2 anni fa

> I want the compiler to scream at me when I write double(“4”) and not let me do it, the guard is doing nothing of the sort.

The optimal solution, then, is for the compiler to be smart enough to recognize that there's a guard and to preemptively enforce it at each callsite. You shouldn't need static typing for this, because the compiler should be able to figure out "oh, there's an is_number guard here, lemme whip up a type constraint for that".

Hell, the even-more-optimal solution would be to not need the guard in the first place, since the * operator already implies an is_number(num) constraint.

__jonas2 anni fa

> The optimal solution, then, is for the compiler to be smart enough to recognize that there's a guard and to preemptively enforce it at each callsite.

It sounds to me like you are describing static typing. As far as I know, type inference from patterns and guards is one of the features of the static type system being developed for Elixir right now [1].

> Hell, the even-more-optimal solution would be to not need the guard in the first place, since the * operator already implies an is_number(num) constraint.

The most optimal solution in my opinion would be a type annotation for the function, so that you do not need to write a guard which adds a runtime overhead just to verify a type that you know you want to always be the same anyways is correct.

Type inference from the `*` operator sounds interesting, unfortunately unlike other languages (like Gleam which has *. +. etc.) Elixir does not have a separate set of operators for floats and integers, so you would not be able to infer which type of number the variable needs to be.

If it were up to me, Elixir would not have pattern matching via functions and instead allowed for type annotation in the function head, I think the matching with multiple function clauses is a lot less readable than just having one function with a big case statement at the top level – of course no way that will change, it is how it is.

[1] https://elixirforum.com/t/full-static-type-inference-of-set-...

yellowapple2 anni fa

> Elixir does not have a separate set of operators for floats and integers, so you would not be able to infer which type of number the variable needs to be.

Elixir doesn't care; it'll convert to a float if necessary (namely, if one of the arguments is a float):

    iex(1)> 1 * 1.0
    1.0

This is identical to the behavior of e.g. Julia:

    julia> 1 * 1.0
    1.0

> The most optimal solution in my opinion would be a type annotation for the function

My point is that the type annotations are redundant, at least in this particular case; the acceptable types are already obvious from the function definition itself.

ccanassa2 anni fa

I love that the creators are considering adding gradual types.

Given TypeScript’s popularity, it’s clear that developers really appreciate types. Honestly, the lack of types is what eventually drove me away from Clojure.

29athrowaway2 anni fa

Erlang is an influential language, it has its uses.

But Erlang and by extension Elixir are a hard sell unless you are writing a system like Whatsapp.

mikhmha2 anni fa

I don't understand the reasoning? I've been working on an MMORPG game since quitting my job - and I settled on using Elixir for this project despite never using it before. I have a good understanding of distributed systems, and the features/tooling Erlang & Elixir provided were like a dream for me. Initially I thought I'd just try making some proof of concept thing. Fast forward several months later, and I seriously credit Elixir for how far I've gotten with this game. Most of my time is spent writing server-side gameplay code, not tracking down obscure networking & memory bugs. Its even caught cascading bugs caused by gameplay system interactions - when players and a.i were being (wrongly) resurrected from the dead the system just crashed! When 500 a.i agents doing pathfinding every single tick was starting to make the a.i system lag and delay inputs - it was trivial to understand the bottleneck causing the system to degrade. I can go on and on.

pdimitar2 anni fa

Please go and on! If you write a blog post I promise you it will have an ardent following and a very warm reception, starting with yours truly.

Snacklive2 anni fa

Please make a blog post it sounds awesome

SoftTalker2 anni fa

Erlang/OTP is the nicest environment for building applications that I've ever experienced. I agree though that it's not popular and I've never actually been paid to use it, with the exception of some small projects that were completely under my control.

mike1o12 anni fa

I disagree, and I think this sentiment is due to a failure in messaging from the Elixir core team and community which is a shame. There is a huge focus on LiveView (which is fantastic) in the Elixir community, but Elixir is so much more than LiveView! Even just writing a simple rest API (or GraphQL through Absinthe) is a great experience, due to libraries like Phoenix and Ecto. It's a joy to work with, even if you aren't writing a distrubuted/realtime system.

I wish there was more of an embrace in the broader ecosystem rather than the focus on LiveView.

doakes2 anni fa

Shouldn't be a hard sell to developers. According to this year's StackOverflow survey, Phoenix is by far the most admired web framework (10% above the second most popular). Elixir is the second most admired language, behind Rust.

https://survey.stackoverflow.co/2024/technology#2-web-framew...

insane_dreamer2 anni fa

The questions in that survey are poorly worded so as to make the results ambiguous, since they confound 1) languages in which X person has worked with in the past year, with 2) languages in which X person would like to work on next year. These express two very different concepts: 1) popularity in the workplace, 2) developers' personal preferences. (Neither of them have to do with the most "admired".

sbuttgereit2 anni fa

I agree. However, the reasoning behind that hard sell has less to do with actual technical considerations and more to do, I believe, with a combination of long held assumptions (not always right assumptions) about what Erlang is and the fact other other stacks have become entrenched and are good enough in the ways that are easiest to assess that Erlang/Elixir would be a tough sell. Add to that recruiting talent to work with Erlang/Elixir requires some outside the box thinking in HR hiring practices (something I rarely accuse HR departments of) and you have a tough sell.

This is all unfortunate. I do think the BEAM based stacks are underrated for applications outside of communications. Many of the same traits of resilience and resource utilization designed to facilitate communications systems actually apply to web apps and APIs, too. Elixir is very expressive and a good fit for writing business applications like accounting related software (what I'm currently working on). But... you have to think you can arbitrage those operational advantages into an overall competitive advantage... and that's a tough sell especially because it involves a lot of speculation which doesn't play out until you're getting to the end of a project.

doawoo2 anni fa

I disagree entirely. Having written everything form little tools for my own use to entire linux based firmware for small embedded machines, Erlang and Elixir are powerful tools. I genuinely find Elixir fun to write in a way no other language manages to match.

insane_dreamer2 anni fa

That's because its inspiration was to create a language with the power of Erlang but that was as enjoyable to code in as Ruby (many people's "most fun to code in" language if they've actually worked with it; unfortunately it can't compete with Python's ecosystem).

klibertp2 anni fa

Erlang and Elixir have a pretty good interop with Python through Erlport. It's not a deep language-level integration (as in, inheriting modules from Python classes or something like that), but it implements a binary message format Erlang nodes use to communicate and offers an API for IPC with Python on top of that. It's not for all use cases, but I had great success deploying a fleet of Raspberries for home automation using nothing but Ansible, Elixir, and Python. Elixir would handle distribution and manage messaging between nodes and local Python processes, while Python scripts would control peripherals. I stopped developing it since I moved to a smaller place, but I firmly believe Elixir+Python is a great solution for anything distributed that would benefit from the Python ecosystem.

insane_dreamer2 anni fa

the big(gest?) advantage that python has are its data science and ML/DL packages

lawik2 anni fa

Heck, I script with Elixir like it was Python.

scop2 anni fa

I build web apps. Basic CRUD stuff with a little bit of business logic sprinkled in. Elixir/Phoenix has been an absolute pleasure to use.

hmmokidk2 anni fa

L take