Functional Core, Imperative Shell (2012)(destroyallsoftware.com)
destroyallsoftware.com
Functional Core, Imperative Shell (2012)
https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell
40 comments
I think that most people, including leads, they do see the benefits, but they think the cons heavily outweighting the pros.
Without having a certain number of key developers pushing the transition it's hard to make this jump.
I would also say that this kind of transition can be had progressively, I myself know a person who pushed a company into a fully-typed static pure functional programming a team that started their whole business in php. Gradually moved them into typed functional php and then into Haskell. He was lucky enough that the team/company was small enough for that at a bigger scale this would've been much more difficult.
In other places I've seen the transition happen (into the functional typescript ecosystem which is a sweet spot imho) it has been mostly the CTO pushing for it.
Without having a certain number of key developers pushing the transition it's hard to make this jump.
I would also say that this kind of transition can be had progressively, I myself know a person who pushed a company into a fully-typed static pure functional programming a team that started their whole business in php. Gradually moved them into typed functional php and then into Haskell. He was lucky enough that the team/company was small enough for that at a bigger scale this would've been much more difficult.
In other places I've seen the transition happen (into the functional typescript ecosystem which is a sweet spot imho) it has been mostly the CTO pushing for it.
How do you gradually move the software of a small company to Haskell? Do you have to build two artifacts, one in PHP and one in Haskell? Two deployment mechanisms?
I don't know the specific details of how the transition went exactly, I didn't work there.
I know they moved into a more functional php, then typed php, then started rewriting some parts of the system to Haskell. The company got eventually bought up by a competitor and killed.
Few companies I know here in Italy are going through a similar into more functional and statically typed php with efforts such as facile.it with libraries like that:
https://github.com/facile-it/php-codec'
Most go through the fp-ts/effect-ts ecosystems and never jump on the scala/haskell languages as they find functional typescript to be the sweet spot for them.
I know they moved into a more functional php, then typed php, then started rewriting some parts of the system to Haskell. The company got eventually bought up by a competitor and killed.
Few companies I know here in Italy are going through a similar into more functional and statically typed php with efforts such as facile.it with libraries like that:
https://github.com/facile-it/php-codec'
Most go through the fp-ts/effect-ts ecosystems and never jump on the scala/haskell languages as they find functional typescript to be the sweet spot for them.
[deleted]
Are these heard con arguments about greenfield development or refactoring proposals?
On greenfield it's worse, because it means from the beginning you have a whole different approach.
On refactoring it's a bit easier, especially since if you choose the right scenario (big new feature), you can just do it that way and then try to teach by example. It's less scary that way, even though there still is initial resistance. Also I think it's better this way, because when you write the tests for the new feature, you can show them side by side with other tests in the codebase. The simplicity of the new tests, and the fewer lines of code but higher coverage, helps sell the deal.
On refactoring it's a bit easier, especially since if you choose the right scenario (big new feature), you can just do it that way and then try to teach by example. It's less scary that way, even though there still is initial resistance. Also I think it's better this way, because when you write the tests for the new feature, you can show them side by side with other tests in the codebase. The simplicity of the new tests, and the fewer lines of code but higher coverage, helps sell the deal.
I think this is a wonderful idea and one of the best ways to structure software, but I want to point out that the functional core is only one way to achieve the benefits and if you are in an environment where that is not possible due to performance or whatever, you can still derive most of the benefits by focusing on the state transformation part.
The real key aspect of this approach is that your state is not spread around the code and changing all over the place, but instead a big State -> State function built out of many smaller referentially transparent functions. (i.e. the complete opposite of OO programming, which IMO is a generally terrible idea that only works well in Smalltalk or in distributed systems).
You can achieve a similar outcome in the procedural setting with a "double buffered" state. Your procedural function writes out a new state based on the previous state, and the "imperative shell" only ever touches the "completed" states, and then flips them so the current state becomes the previous state and the previous state becomes the buffer to have the new state written into.
Less convenient but if you need to have tight control over memory allocations or performance this can be beneficial.
The real key aspect of this approach is that your state is not spread around the code and changing all over the place, but instead a big State -> State function built out of many smaller referentially transparent functions. (i.e. the complete opposite of OO programming, which IMO is a generally terrible idea that only works well in Smalltalk or in distributed systems).
You can achieve a similar outcome in the procedural setting with a "double buffered" state. Your procedural function writes out a new state based on the previous state, and the "imperative shell" only ever touches the "completed" states, and then flips them so the current state becomes the previous state and the previous state becomes the buffer to have the new state written into.
Less convenient but if you need to have tight control over memory allocations or performance this can be beneficial.
The issue is that while this greatly simplifies the program logic, when you start to scale the application you do end up with a lot of redundant computation. So in the end you need to either:
. take a performance hit
. somehow track/update costly intermediaries in your global state, and this can balloon out of control really fast
. pair it with some memoization mechanism/framework that automatically manages "derived state" variables
. take a performance hit
. somehow track/update costly intermediaries in your global state, and this can balloon out of control really fast
. pair it with some memoization mechanism/framework that automatically manages "derived state" variables
Yup, and I strongly suggest option 1, just take the perf loss. You can probably detect in each of your state update functions that nothing changed and just reuse the previous result (a memcpy is pretty fast, or if that data is in a separate allocation just copy the pointer).
The same is true of the functional core idea really, you're still going to have a big tree of function calls rebuilding things unless you also add some memoization to it. React and friends do it because DOM manipulation is very slow.
But think of IMGUI style GUIs, they're rebuilding the entire user interface every frame and yet are often more efficient than many retained mode UIs. Redundant computation is not as bad as it sounds at first glance.
The same is true of the functional core idea really, you're still going to have a big tree of function calls rebuilding things unless you also add some memoization to it. React and friends do it because DOM manipulation is very slow.
But think of IMGUI style GUIs, they're rebuilding the entire user interface every frame and yet are often more efficient than many retained mode UIs. Redundant computation is not as bad as it sounds at first glance.
I asked around and there is another option. Something called "Incremental Computations". Here is a Clojure library. I think the examples kinda demo how it works
https://github.com/hoplon/javelin
It's sort of a version of option 3
https://github.com/hoplon/javelin
It's sort of a version of option 3
> and yet are often more efficient than many retained mode UIs
I don’t know what metric you use for that, but they are definitely not more energy efficient — retained mode is a must for any battery-powered device.
I don’t know what metric you use for that, but they are definitely not more energy efficient — retained mode is a must for any battery-powered device.
I meant computational efficiency, highly dynamic UIs usually perform a lot better as immediate mode GUIs since there are less indirections and memory allocations going around.
If you're rebuilding things a lot anyway, it's actually more efficient to just accept that rather than work around it.
Though speaking of immediate mode UIs, I think a lot of the problem there is that most of the immediate mode UI libraries assume they're being used as the user interface of a Game or 3D render application which needs to render a complex 3D scene regardless, so who cares about the extra 400 microseconds the UI takes to render.
You could imagine optimizing immediate mode UIs quite a bit for energy efficiency by not redrawing anything if there's no use input, or selectively redrawing dirty areas where it makes sense.
If you're rebuilding things a lot anyway, it's actually more efficient to just accept that rather than work around it.
Though speaking of immediate mode UIs, I think a lot of the problem there is that most of the immediate mode UI libraries assume they're being used as the user interface of a Game or 3D render application which needs to render a complex 3D scene regardless, so who cares about the extra 400 microseconds the UI takes to render.
You could imagine optimizing immediate mode UIs quite a bit for energy efficiency by not redrawing anything if there's no use input, or selectively redrawing dirty areas where it makes sense.
> You could imagine optimizing immediate mode UIs quite a bit for energy efficiency by not redrawing anything if there's no use input, or selectively redrawing dirty areas where it makes sense.
That’s pretty much what a retained-mode UI does.
That’s pretty much what a retained-mode UI does.
I don't mean doing it in a retained-mode-like way with a data structure in memory, I mean special casing certain UI elements. For example, a blinking cursor could be a special path through the code which updates only the pixels where the blinking cursor is, while the rest of the window just reuses the last drawn frame (kinda like how mouse cursors are handled in hardware). Similar tricks work for things like hover highlights and context menus.
No need for a retained data structure to implement the above.
No need for a retained data structure to implement the above.
On the other hand, I've found that a lot of the caching that happens organically from each component building up it's own state of the world can lead to poor performance of its own. You often end up reading single values from random places in memory.
You can often optimize a single, large state transformation by utilizing that it does a lot of similiar work. You can also often get a big performance boost by batching up computations.
Half of all performance problems are solved by introducing a cache. The other half is solved by removing one.
You can often optimize a single, large state transformation by utilizing that it does a lot of similiar work. You can also often get a big performance boost by batching up computations.
Half of all performance problems are solved by introducing a cache. The other half is solved by removing one.
Conceptually this reminds me a bit of a ring buffer, which also deals with the issue of having to manage a fixed size memory allocation.
Or a draft/publish workflow where you’re constantly re-use memory instead of remembering published documents.
I like it! It’s clever and seems to be generally applicable. Is there a name for it?
Or a draft/publish workflow where you’re constantly re-use memory instead of remembering published documents.
I like it! It’s clever and seems to be generally applicable. Is there a name for it?
There's probably a name, but I don't know it, I saw the idea somewhere before but can't remember where.
I think Gary Bernhardt's corresponding Boundaries conference talk [1] is quite good, particularly because of how pragmatic he is in introducing the topic.
I have been thinking about and trying to work with this paradigm for a few years now. To be honest, it's been a struggle. Partly because it requires unlearning some habits and partly because there's limited opportunity to practice in my $DAYJOB where I'm working in codebases heavily invested in other paradigms.
I've seen and read many, many videos, blog posts, etc. on this topic. Everyone loves to introduce the concept with a contrived example and then stop there. What I would love to see more of are complete examples for everything from simple to complex applications.
I think there's potentially a lot of value in thinking this way (or at least taking as much from it as you can), but it's hard to learn and then to teach others without more examples to draw on.
[1]: https://www.destroyallsoftware.com/talks/boundaries
I have been thinking about and trying to work with this paradigm for a few years now. To be honest, it's been a struggle. Partly because it requires unlearning some habits and partly because there's limited opportunity to practice in my $DAYJOB where I'm working in codebases heavily invested in other paradigms.
I've seen and read many, many videos, blog posts, etc. on this topic. Everyone loves to introduce the concept with a contrived example and then stop there. What I would love to see more of are complete examples for everything from simple to complex applications.
I think there's potentially a lot of value in thinking this way (or at least taking as much from it as you can), but it's hard to learn and then to teach others without more examples to draw on.
[1]: https://www.destroyallsoftware.com/talks/boundaries
I've used this for many real applications that I would characterise as small (biggest being ~10KLOC excluding tests, types, etc.). I've applied the pattern across entire codebases and to specific components within much larger codebases which follow other patterns. I tend to combine it with some of the ideas in Mitchell Hashimoto's talk Advanced Testing with Go[0] - particularly small interfaces and defining test stubs alongside real implementations.
In practice my imperative shell tends to have two layers. The inner layer is responsible for executing the imperative logic, while the outer layer is responsible for initialising configuration and dependencies, invoking the inner layer, and adhering to any sort of external interface that it may need to satisfy. Everything from the inner layer down through the functional core can be comprehensively tested using stub objects only -- no need to patch anything.
Unfortunately everything I've applied this pattern to is proprietary, so I can't share any code examples.
[0] https://www.youtube.com/watch?v=yszygk1cpEc
In practice my imperative shell tends to have two layers. The inner layer is responsible for executing the imperative logic, while the outer layer is responsible for initialising configuration and dependencies, invoking the inner layer, and adhering to any sort of external interface that it may need to satisfy. Everything from the inner layer down through the functional core can be comprehensively tested using stub objects only -- no need to patch anything.
Unfortunately everything I've applied this pattern to is proprietary, so I can't share any code examples.
[0] https://www.youtube.com/watch?v=yszygk1cpEc
I also did it for a Go app recently exactly like that.
It was mostly about putting the "imperative" part closer to main.go and putting business rules and things like that in the other files: Routing, serialization, validation, business rules, data transformation, command line argument parsing, configuration parsing, they all go in the "functional" part. Instantiating the HTTP server, reading argv, files and reading/writing to the database goes in the "imperative" part.
The major hurdle for me is frameworks and libraries that often want to be used in an imperative way. Some routers, for, instance could be purely functional, but they often want to instantiate the HTTP server themselves. Not a big issue in practice in Golang, though.
Also: the database. To keep purity in a simple way, you gotta wrap the imperative parts and use callback-ish structures. This part is often where I cheat. But with a proper abstraction it's great. I hate to say the forbidden word here, but a "monadic" abstraction you can have your cake and eat it. I never did it in Golang and I have no idea whether it would work, but, I did it in C# and it was a breeze.
I agree with your assessment about testing. In the end it was incredibly easy for me to get 100% coverage on the functional part.
It was mostly about putting the "imperative" part closer to main.go and putting business rules and things like that in the other files: Routing, serialization, validation, business rules, data transformation, command line argument parsing, configuration parsing, they all go in the "functional" part. Instantiating the HTTP server, reading argv, files and reading/writing to the database goes in the "imperative" part.
The major hurdle for me is frameworks and libraries that often want to be used in an imperative way. Some routers, for, instance could be purely functional, but they often want to instantiate the HTTP server themselves. Not a big issue in practice in Golang, though.
Also: the database. To keep purity in a simple way, you gotta wrap the imperative parts and use callback-ish structures. This part is often where I cheat. But with a proper abstraction it's great. I hate to say the forbidden word here, but a "monadic" abstraction you can have your cake and eat it. I never did it in Golang and I have no idea whether it would work, but, I did it in C# and it was a breeze.
I agree with your assessment about testing. In the end it was incredibly easy for me to get 100% coverage on the functional part.
Would love to see an example of your monadic database abstraction
If you're familiar with Node.js promises, you already know how it works. I just pass a hollow "DB" object to my controllers that can perform queries and commands, but instead of returning data imperatively, I use JS-like "promises" to chain the next steps. On the "success" callback you have the result of queries. At the end I will just return this "chain of statements" that wasn't executed yet. I only really run everything at the "imperative" layer, within a database transaction.
A big issue is that this was in very small project. Promises are not exactly pretty, and the code is not the easiest to maintain if you don't know how they work, which is why I "cheated" on the Golang app. I think someone else smarter might be able to figure this problem out too, though :)
return DB.Query(...).ThenCommand(res => ...).ThenHttpSuccess(res => ...);
A big issue is that this was in very small project. Promises are not exactly pretty, and the code is not the easiest to maintain if you don't know how they work, which is why I "cheated" on the Golang app. I think someone else smarter might be able to figure this problem out too, though :)
> In practice my imperative shell tends to have two layers.
Doesn't that boil down to a hexagonal design, ports & adapters, that is also mentioned at the end of the Boundaries video?
Doesn't that boil down to a hexagonal design, ports & adapters, that is also mentioned at the end of the Boundaries video?
I'm not super up to date with the web-dev world - but I think the folks in the react-and-company space have taken this the natural step forward
With subscription/contexts you get to really pare down the whole imperative shell part. They allow you to keep using reproducible pure functions, but memoization allows you drastically minimize state management and to cache intermediary values, so you're not recomputing the whole world each time. When the state changes the whole system elegantly only recomputes the pieces that are strictly necessary. You don't need to do anything manually - it all just happens automatically
For some reason I don't see this being picked up in generally outside of GUI/Web space. But the input side of state is semi-solved as I see it
The output side still needs "management" or a "shell" of sorts
(for context, I've only really played with this through Clojure's cljfx GUI library)
With subscription/contexts you get to really pare down the whole imperative shell part. They allow you to keep using reproducible pure functions, but memoization allows you drastically minimize state management and to cache intermediary values, so you're not recomputing the whole world each time. When the state changes the whole system elegantly only recomputes the pieces that are strictly necessary. You don't need to do anything manually - it all just happens automatically
For some reason I don't see this being picked up in generally outside of GUI/Web space. But the input side of state is semi-solved as I see it
The output side still needs "management" or a "shell" of sorts
(for context, I've only really played with this through Clojure's cljfx GUI library)
This is a classic and one of the most insightful talks/paradigms I've encountered. One of the few that stands the test of many years.
People get confused by the wizardly and can't recognize that in many places they are already doing it. A resource orientated REST API is a great example of this paradigm. The API model is declarative... but you interface with it as sequential http calls.
So just do that in code signatures too.
So just do that in code signatures too.
Almost the opposite of react, which is imperative core functional shell.
Could you elaborate? Do you mean the implementation internals of React, or how dev's end up using the React library?
I tend to agree with https://svitla.com/blog/functional-programming-in-typescript....:
> Enter the functional core, imperative shell approach where you can do as much work as possible in a functional way, and isolate impure interactions on a thin, outer layer.
> The most prominent example of this approach is ReactJS where:
> - All components can be seen as pure functions that take props, state, and context as an input and return a React element as an output.
> - All “outside world interactions” take place on the ReactDOM render/hydrate function.
I tend to agree with https://svitla.com/blog/functional-programming-in-typescript....:
> Enter the functional core, imperative shell approach where you can do as much work as possible in a functional way, and isolate impure interactions on a thin, outer layer.
> The most prominent example of this approach is ReactJS where:
> - All components can be seen as pure functions that take props, state, and context as an input and return a React element as an output.
> - All “outside world interactions” take place on the ReactDOM render/hydrate function.
If you try and put all your state at the top and make the rendering a function of it, you end up with excessive re-renders. So you end up co-locating state at your leaf components - hidden state everywhere.
> This design has many nice side effects.
Hahaha...great choice of words for an article on functional programming =)
Hahaha...great choice of words for an article on functional programming =)
Previous discussion:
- https://hw.leftium.com/#/item/18043058 (127 comments)
- https://hw.leftium.com/#/item/29613107
---
A FAQ is "Are there example of non trivial apps written in this manner?"
- CodeMirror/ProseMirror [1]
- React [2]
[1]: https://codemirror.net/docs/guide/#:~:text=Functional%20Core...).
[2]: https://svitla.com/blog/functional-programming-in-typescript....
- https://hw.leftium.com/#/item/18043058 (127 comments)
- https://hw.leftium.com/#/item/29613107
---
A FAQ is "Are there example of non trivial apps written in this manner?"
- CodeMirror/ProseMirror [1]
- React [2]
[1]: https://codemirror.net/docs/guide/#:~:text=Functional%20Core...).
[2]: https://svitla.com/blog/functional-programming-in-typescript....
Or, to keep things on HN:
- https://news.ycombinator.com/item?id=18043058
- https://news.ycombinator.com/item?id=29613107
- https://news.ycombinator.com/item?id=18043058
- https://news.ycombinator.com/item?id=29613107
This is much like the Elm Architecture [1], a pattern that I enjoy using. That structure gives me problems though when I want to speed things up by caching frequent calculations and lookups in the View functions. In Elm, at least, this is only possible by keeping the cache explicitly in the Model and maintaining it in the Update function. So it feels like considerations specific to generating a particular view end up explicitly in the more general update of the application-global Model. And for the Update to avoid doing lots of such view-value computations not needed by the current view, the Update function has to know a lot about view-only state.
[1]: https://guide.elm-lang.org/architecture/
[1]: https://guide.elm-lang.org/architecture/
Isn't this what we all are doing since the "birth" of the so-called web frameworks?
The funtional core is mainly your "use cases"/"services": they the core of your software because they deal with your domain model. They are functional in that they do not keep state (state belongs to the DB).
The imperative shell is all about the "web controllers" and "db concrete implementations", and whatever i/o is needed. So, in your controller you read your POST request and retrieve the params you wanna work with; you also handle the rendering of templates if needed. Your db concrete implementation is all about SQL (or ORMs).
The funtional core is mainly your "use cases"/"services": they the core of your software because they deal with your domain model. They are functional in that they do not keep state (state belongs to the DB).
The imperative shell is all about the "web controllers" and "db concrete implementations", and whatever i/o is needed. So, in your controller you read your POST request and retrieve the params you wanna work with; you also handle the rendering of templates if needed. Your db concrete implementation is all about SQL (or ORMs).
Am kind of using this at work. All IO is isolated in an imperative shell layer, that communicates with an core via event. The core is kind of a state transformer, but actually does in place mutations. All state (of the business logic) is kept at one place, which has some nice benefits.
Functional programming is not too important in the core, a lot of benefits arise without it, e.g.: I can replay a prod session from the events generated and get the exact same behaviour. Makes debugging a breeze.
One advantage of having the state in one big place is that I can serialize it, send it to another thread that serves a UI from it.
Functional programming is not too important in the core, a lot of benefits arise without it, e.g.: I can replay a prod session from the events generated and get the exact same behaviour. Makes debugging a breeze.
One advantage of having the state in one big place is that I can serialize it, send it to another thread that serves a UI from it.
If it helps anyone:
I find it typically easier to ask “can I extract data and functions out of here” rather than “can I extract the side-effects out of here”. And I found that asking this question too early can be detrimental if you’re still trying to figure stuff out.
I don’t follow the paradigm strictly and I often write code in order to understand something. The first draft is typically just procedural with minimal abstractions. It’s more efficient that way.
Only after it’s working I start to pull out the pure bits and write tests for them.
I find it typically easier to ask “can I extract data and functions out of here” rather than “can I extract the side-effects out of here”. And I found that asking this question too early can be detrimental if you’re still trying to figure stuff out.
I don’t follow the paradigm strictly and I often write code in order to understand something. The first draft is typically just procedural with minimal abstractions. It’s more efficient that way.
Only after it’s working I start to pull out the pure bits and write tests for them.
That's actually exactly how the author (Gary Bernhardt) describes implementing Functional Core, Imperative Shell:
> This is the cadence of the entire project so far. I tend to bloat up this outer file to something like 250 lines in some cases. Then I extract behavior out, moving it into unit-tested classes, figuring out how to make it more functional.
Skip to about 11:40 in the video. (There is some additional explanation right before that.)
> This is the cadence of the entire project so far. I tend to bloat up this outer file to something like 250 lines in some cases. Then I extract behavior out, moving it into unit-tested classes, figuring out how to make it more functional.
Skip to about 11:40 in the video. (There is some additional explanation right before that.)
Video doesn't work for me in Firefox, but if it is the one I think it is, I have seen it and it is very good. Recommended.
Works fine for me on Firefox 110, macOS Monterey 12.6.3, M1 Pro. Perhaps it's an extension?
I find both of these scenarios disappointing and frustrating, because technology (and software) is usually about improving processes and systems so better results can be made more easily. But at this point it feels like a lost cause, or rather a hill not worth dying on. For my own projects, I go this path; for others, I work dumber and keep the corporate apple cart upright.
Rant aside, there is a video which I really like that demonstrates this functional core/imperative shell approach: Solving Problems the Clojure Way - Rafal Dittwald. It is a Clojure talk, but it gives an imperative to functional refactor demo in JavaScript in the middle of the talk. Even though it is a toy example, it is big enough that the result shows a meaningful difference in approach.
https://youtu.be/vK1DazRK_a0