Meta Programming in JavaScript with Proxies(itnext.io)
itnext.io
Meta Programming in JavaScript with Proxies
https://itnext.io/meta-programming-in-javascript-with-proxies-64fa4898070e
54 コメント
I don't know if it's a subtle joke, but I think you have a typo in readme.md s/pointdexter/poindexter.
Yea, Proxy performance can be an issue, especially if you use them recursively. I tried using them to create a "protected" clone (https://github.com/goodoldneon/alhambra). The initial "clone" is much more performant than actually deep cloning, and doesn't get more expensive as the size of the original increases. However, get/set/delete/etc. is far less performant.
> I do worry about their effects on performance
What kind of performance do you mean? Compile time? They should be optimisable away and shouldn't have any effect on actual run time should they?
What kind of performance do you mean? Compile time? They should be optimisable away and shouldn't have any effect on actual run time should they?
Proxies are entirely a runtime API. Did you read the article?
> Proxies are entirely a runtime API.
Yeah I know - but where isn't the performance as good as it should be? Is it during compile time, runtime, memory, something else? Do they not get compiled them away so they're free in terms of runtime cost?
> Did you read the article?
Yes of course I read the article. It's against HN guidelines to ask people this.
Yeah I know - but where isn't the performance as good as it should be? Is it during compile time, runtime, memory, something else? Do they not get compiled them away so they're free in terms of runtime cost?
> Did you read the article?
Yes of course I read the article. It's against HN guidelines to ask people this.
They don't get compiled away because Proxies are a runtime thing. The impact on performance is entirely on the runtime.
I have no idea what you are trying to say. I know they're a runtime thing. Why is it not possible to optimise them away so they don't have a runtime cost, after having been compiled?
Because they need runtime support from the engine, e.g. you need engine support in order to intercept calls like:
obj[prop] = 'foo'
set(obj, prop, "foo")
get(obj, prop)
I don't mean Babel source-to-source compilation. I mean the native compilation done at runtime. The JIT.
There's some work being done on optimising Proxies, but the current implementation is just slow (and also varies by engine), but yes, they can probably be optimised further.
EDIT: there are various blog posts around talking about optimisation: https://v8.dev/blog/optimizing-proxies
EDIT: there are various blog posts around talking about optimisation: https://v8.dev/blog/optimizing-proxies
It's not possible to pre-compile away the dynamic part of a Proxy's behavior.
For instance, if you were to receive a random string over some network call and set that on a proxified object, how would you pre-compile the Proxy away such that its runtime behavior is preserved when accessing what could be any arbitrary string?
You'd need infinite space to represent the space of all possible string inputs in order to optimize that away at compile time, which is obviously not feasible.
For instance, if you were to receive a random string over some network call and set that on a proxified object, how would you pre-compile the Proxy away such that its runtime behavior is preserved when accessing what could be any arbitrary string?
You'd need infinite space to represent the space of all possible string inputs in order to optimize that away at compile time, which is obviously not feasible.
I know you can't AOT compile it away, that seems obvious, but what's the limitation that means you can't JIT compile it away? Like any other JS optimisation such as inline caching?
JIT isn't really compiling the runtime performance penalty of the Proxy "away" though.
JIT compilation still involves a runtime performance penalty of the JIT compilation step itself, that has to be traded off against the runtime performance penalty of just running the Proxy logic without any runtime compilation step.
That's a nuanced tradeoff that has to be weighed on a case by case basis by various heuristics, and can reduce the runtime cost of proxies when applied to the right cases (and at the same time, could increase runtime cost if applied to the wrong ones: think overly aggressive inlining and its effects on memory usage), but either way it doesn't result in the Proxy abstraction becoming "compiled away" into something that has 0 runtime cost.
JIT compilation still involves a runtime performance penalty of the JIT compilation step itself, that has to be traded off against the runtime performance penalty of just running the Proxy logic without any runtime compilation step.
That's a nuanced tradeoff that has to be weighed on a case by case basis by various heuristics, and can reduce the runtime cost of proxies when applied to the right cases (and at the same time, could increase runtime cost if applied to the wrong ones: think overly aggressive inlining and its effects on memory usage), but either way it doesn't result in the Proxy abstraction becoming "compiled away" into something that has 0 runtime cost.
[deleted]
> Unfortunately, to get the last element we still have to do the clunky javaScriptArray[ javaScriptArray.length - 1 ] technique. This ... has always seemed like a major oversight in the ECMAScript standard.
Sort of an aside, but there's a stage 1 (relatively early) proposal to add an `array.lastItem` property that should finally improve things:
https://github.com/keithamus/proposal-array-last
Sort of an aside, but there's a stage 1 (relatively early) proposal to add an `array.lastItem` property that should finally improve things:
https://github.com/keithamus/proposal-array-last
In the list of useful functions I'd accumulated over a few years of JS, there are `[].last` and `[].lastIndex()`.
The former is a property getter/setter implanted into the Array object¹. The latter is a function that returns `[].length - 1 || 0`.
¹ I know planting properties and functions directly onto the basic objects is generally frowned upon, but I see no harm in doing that for personal projects. Using it in the big-web production might be verboten, but it doesn't I can't make my coding life a bit easier.
Meanwhile, I have a problem with the "web compatibility naming safeguard" argument over the names of the proposed properties/functions. Surely, the websites using `[].last` override the namespace to begin with. If, say, `[].last` were to be accepted, how could it break the sites using the override version?
Outside of that, the fact that new proposals have to cow before a widespread conventional use of the names renders me confused. I feel like the JS engine should take priority over mere convention. Sure, some websites might be hurt in the process, but at least the generations of current and future coders would have few problems figuring out what the new functions do.
Exhibit 1: `globalThis`. Nuff said?
Exhibit 2: `[].smoosh`, which apparently was in the books [https://github.com/staltz/prevent-smoosh]. No longer is: it's now `[].flat`.
Exhibit 3: `Math.clamp()`, in questions [https://github.com/rwaldron/proposal-math-extensions#questio...] for the `Math` extension proposal. (Yet undecided-upon.)
`[].last` just makes sense.
The former is a property getter/setter implanted into the Array object¹. The latter is a function that returns `[].length - 1 || 0`.
¹ I know planting properties and functions directly onto the basic objects is generally frowned upon, but I see no harm in doing that for personal projects. Using it in the big-web production might be verboten, but it doesn't I can't make my coding life a bit easier.
Meanwhile, I have a problem with the "web compatibility naming safeguard" argument over the names of the proposed properties/functions. Surely, the websites using `[].last` override the namespace to begin with. If, say, `[].last` were to be accepted, how could it break the sites using the override version?
Outside of that, the fact that new proposals have to cow before a widespread conventional use of the names renders me confused. I feel like the JS engine should take priority over mere convention. Sure, some websites might be hurt in the process, but at least the generations of current and future coders would have few problems figuring out what the new functions do.
Exhibit 1: `globalThis`. Nuff said?
Exhibit 2: `[].smoosh`, which apparently was in the books [https://github.com/staltz/prevent-smoosh]. No longer is: it's now `[].flat`.
Exhibit 3: `Math.clamp()`, in questions [https://github.com/rwaldron/proposal-math-extensions#questio...] for the `Math` extension proposal. (Yet undecided-upon.)
`[].last` just makes sense.
I miss python’s list operators so much when I’m in JS (along with lots of other stuff). Is there a reason why we couldn’t use python’s x[-1] syntax in JS?
There's probably existing code out there that relies on `arr[-1]` returning `undefined`, e.g. looping backward through an array until it gets a falsy value. You can also assign to `arr[-1]` and it'll work (it assigns to a new -1 key, not to the end of the array), and maybe some people do that in existing code. They had to pick `lastItem` as the name instead of just `last` since using `last` is also known to break the web.
It's also not obvious that negative array accesses should behave that way. It's a tradeoff where it's more convenient in some cases but feels a bit less elegant and can lead to bugs (particularly dynamic array accesses that are accidentally negative). But FWIW, `slice` already implements this behavior, e.g. `arr.slice(-1)[0]` is a way to get the last element of an array, so there's certainly precedent in JS for it.
It's also not obvious that negative array accesses should behave that way. It's a tradeoff where it's more convenient in some cases but feels a bit less elegant and can lead to bugs (particularly dynamic array accesses that are accidentally negative). But FWIW, `slice` already implements this behavior, e.g. `arr.slice(-1)[0]` is a way to get the last element of an array, so there's certainly precedent in JS for it.
I use these sometimes in personal projects:
https://gist.github.com/thisredone/783407f9aacc496c4c13e4076...
https://gist.github.com/thisredone/783407f9aacc496c4c13e4076...
All the examples could be done in ES5 using Object.defineProperty. Proxies do however have more traps available.
Yep, proxies can be really useful, but use them carefully. We use them at Yazz so that we can have custom code editors that have variables injected into them
> get(tgt, prop, rcvr)
Good article, terrible variable naming.
Abbreviated variables are a pet peeve of mine. What is 'tgt' supposed to be? An abbreviation of 'target'? 'Thank god it's Thursday'? Might as well just use x,y and z, it's no less clear.
Good article, terrible variable naming.
Abbreviated variables are a pet peeve of mine. What is 'tgt' supposed to be? An abbreviation of 'target'? 'Thank god it's Thursday'? Might as well just use x,y and z, it's no less clear.
funny, target, propery, receiver seem evident to me. don't you guys do note-taking ?
What is the convention used here for deciding the abbreviated names? It's clearly not removing vowels since prop has an o in it. So it becomes subjective as to how stuff should be abbreviated. If things are subjective, there is no standard.
In one function developer A might call it "rcvr", in another function developer B might call it "rec". Both mean "receiver", just use "receiver".
In one function developer A might call it "rcvr", in another function developer B might call it "rec". Both mean "receiver", just use "receiver".
`rcvr` is clearly Resistor Capacitor Viewer. Duh. And clearly `rec` is Recovery.
The Story of Bunny Showers:
I worked on a code base where we had, in one class, seemingly endless variations on the spelling of "business hours". There were variables named `bsns_hrs`, `business_hours`, ` business_hrs`. Furthermore, in the object's (horrible, messy, overlarge, mutable as fsck) internal state, there were elements labeled with `busnessHours`, `bsnsHours`, 'business-hours`, 'bnshrs`. And so on.
My habbit when reading code like this is to map each spelling to a phonetically similar word or phrase. So `business_hrs` becomes "business hers" and `bnshrs` becomes "bunny showers". And so forth. It makes it easier to rationalize about what is happening, especially if, like me, you move your lips when you read.
I have ever since called this issue of inconsistent spelling in code the "bunny showers" problem, owing both to the well-known fecundity of rabbits and the hellish proliferation of inconsistent names in that fabled codebase.
The Story of Bunny Showers:
I worked on a code base where we had, in one class, seemingly endless variations on the spelling of "business hours". There were variables named `bsns_hrs`, `business_hours`, ` business_hrs`. Furthermore, in the object's (horrible, messy, overlarge, mutable as fsck) internal state, there were elements labeled with `busnessHours`, `bsnsHours`, 'business-hours`, 'bnshrs`. And so on.
My habbit when reading code like this is to map each spelling to a phonetically similar word or phrase. So `business_hrs` becomes "business hers" and `bnshrs` becomes "bunny showers". And so forth. It makes it easier to rationalize about what is happening, especially if, like me, you move your lips when you read.
I have ever since called this issue of inconsistent spelling in code the "bunny showers" problem, owing both to the well-known fecundity of rabbits and the hellish proliferation of inconsistent names in that fabled codebase.
“Receiver” should doubtless be abbreviated to “rx” (common in telecommunications, along with “tx” for “transmitter”).
(This has the added bonus of potential confusion with RxJS.)
(This has the added bonus of potential confusion with RxJS.)
[deleted]
Exactly this. I take a hard stance against abbreviation in our team's codebases not solely because people might have trouble understanding what they mean (though when you take abbreviation to the extreme, i.e. words to single letters, that becomes a huge problem as well), but also because of the tendency for abbreviation to lead to an explosion in different slight variations of the same name used across the codebase to represent the same things, which increases the chances of missing important pieces in a refactor and decreases the chances of someone being able to find whatever they might be looking for.
Keeping naming conventions consistent across a codebase with multiple people working on it is hard enough to begin with. Abbreviation makes it even harder. Definitely not worth the few keystrokes saved, imho.
Keeping naming conventions consistent across a codebase with multiple people working on it is hard enough to begin with. Abbreviation makes it even harder. Definitely not worth the few keystrokes saved, imho.
From reading many blogs my intuition is that the standard is "four letter or less that sound kind of similar when read"...
Having experimented with and used proxies a decent amount, the MDN page is my goto reference when I need to use them: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
The biggest pitfalls, in my experience are
1. Infinite recursion when implementing get and printing proxies with console.log in node.js (which attempts to access properties from the object prototype which can lead to infinite recursion problems).
2. Performance, which I would need to do more digging and benchmarking to comment on in more depth.
Some libraries that use Proxies to great effect are the https://github.com/mobxjs/mobx-state-tree and https://github.com/mweststrate/immer libraries both by mweststrate
The biggest pitfalls, in my experience are
1. Infinite recursion when implementing get and printing proxies with console.log in node.js (which attempts to access properties from the object prototype which can lead to infinite recursion problems).
2. Performance, which I would need to do more digging and benchmarking to comment on in more depth.
Some libraries that use Proxies to great effect are the https://github.com/mobxjs/mobx-state-tree and https://github.com/mweststrate/immer libraries both by mweststrate
Immer is excellent. It makes deeply nested immutable updates a breeze. The API almost makes it feel like you're programming in a fantasy-land future where JavaScript's default data structures were persistent data structures like a proper functional language.
And it accomplishes all that without sacrificing type safety like most string path based approaches to deep updates often tend to do (think lodash.set or ramda.assocPath).
The only reason we haven't gone all in on it yet is the existence of browsers that don't support Proxy (like IE11, which we now unfortunately have to support), where performance takes a huge hit, and those tend to be the browsers that can least afford to take that performance hit to begin with.
And it accomplishes all that without sacrificing type safety like most string path based approaches to deep updates often tend to do (think lodash.set or ramda.assocPath).
The only reason we haven't gone all in on it yet is the existence of browsers that don't support Proxy (like IE11, which we now unfortunately have to support), where performance takes a huge hit, and those tend to be the browsers that can least afford to take that performance hit to begin with.
Yep, we're using Immer in our new Redux Starter Kit package to help make writing Redux reducers easy:
https://redux-starter-kit.js.org
https://redux-starter-kit.js.org
I just converted one of our medium sized redux apps to immer and it has been a resounding success. I highly recommend it and will be using it everywhere in the future.
Hi, I'm a Redux maintainer. I'd be interested in hearing how your experience with Immer went, as well as any other feedback you can offer.
Also, as I mentioned just below, please try out our new Redux Starter Kit package, which uses Immer internally (and offers several other useful utilities for simplifying common Redux use cases, like store setup, reducer definitions, and even creating entire "slices" of state at once):
https://redux-starter-kit.js.org
Also, as I mentioned just below, please try out our new Redux Starter Kit package, which uses Immer internally (and offers several other useful utilities for simplifying common Redux use cases, like store setup, reducer definitions, and even creating entire "slices" of state at once):
https://redux-starter-kit.js.org
Actually the reason I was willing to take a chance on Immer is because it's included in redux starter kit. :)
Using immer in production has been about 1000x better than writing immutable state updates by hand. It literally reduced (hah) the LOC in our reducers by at least 75%, not to mention the gains from reduced cognitive overhead.
The architecture I ended up using is basically a recreation of redux starter kit with some customization for convenience (a couple of methods on action creators that wrap `toString` for creating reducer keys that only handle actions with certain meta properties, like `updateEntities.of(entity)`).
Using immer in production has been about 1000x better than writing immutable state updates by hand. It literally reduced (hah) the LOC in our reducers by at least 75%, not to mention the gains from reduced cognitive overhead.
The architecture I ended up using is basically a recreation of redux starter kit with some customization for convenience (a couple of methods on action creators that wrap `toString` for creating reducer keys that only handle actions with certain meta properties, like `updateEntities.of(entity)`).
Neat! I'd be interested in seeing what you came up with if you get a chance.
I rewrote my reducers with Immer as well. I had some pretty complex nested updates that became super simple with Immer, love it!
An interesting article, but I would be concerned about readability for newcomers unfamiliar with Proxies.
To be honest, for the particular use cases mentioned here - why not use the object utilities in Lodash (which is well used, familiar to many, well documented, and battle hardened)?
> we didn’t get around to implementing slices like array[1:3]
What's wrong with arr.slice(1,3)?
To be honest, for the particular use cases mentioned here - why not use the object utilities in Lodash (which is well used, familiar to many, well documented, and battle hardened)?
> we didn’t get around to implementing slices like array[1:3]
What's wrong with arr.slice(1,3)?
This rubs me the wrong way somehow.
All of this functionality could be accomplished with simple functions. They may not be as cool, but they are obvious in the calling code and they are easy to search for. Magic stuff like this conflicts with community conventions. New people are going to be perplexed as to what's going on (or worse, they learn from your code and conversely get perplexed when x[-1] doesn't work on a normal array). Also, you risk confusing various tools in subtle ways.
All of this functionality could be accomplished with simple functions. They may not be as cool, but they are obvious in the calling code and they are easy to search for. Magic stuff like this conflicts with community conventions. New people are going to be perplexed as to what's going on (or worse, they learn from your code and conversely get perplexed when x[-1] doesn't work on a normal array). Also, you risk confusing various tools in subtle ways.
I agree. While you can accomplish some really interesting things with Proxy, I think that this article presents some poor examples that are actively harmful. I don't expect `delete` to be O(N) (actually worse if you consider the `ownKeys` implementation. I don't expect setting a property (sort) to invoke an expensive sorting operation. The list keeps going.
While I appreciate what the author is trying to accomplish, I think this is an example of what not to do as far as code is concerned.
Most of my practical usage of Proxies to date have been limited to some debugging tools that I wrote for AngularJS to help detect property changes that occur outside of a digest cycle. Don't think I've had any use cases that I've shipped to production code yet, though I believe Vue is considering using proxies for their reactivity model in the future to avoid the need for `Vue.get` and `Vue.set` for previous unknown properties or deletions.
While I appreciate what the author is trying to accomplish, I think this is an example of what not to do as far as code is concerned.
Most of my practical usage of Proxies to date have been limited to some debugging tools that I wrote for AngularJS to help detect property changes that occur outside of a digest cycle. Don't think I've had any use cases that I've shipped to production code yet, though I believe Vue is considering using proxies for their reactivity model in the future to avoid the need for `Vue.get` and `Vue.set` for previous unknown properties or deletions.
I particularly like the ``JSON.stringify({ name: "Albert Einstein" }) in dbProxy``.
I could see it being extremely useful for inspecting and debugging, but I shudder to think of working on a large codebase with liberal use of proxies for production behavior. You wouldn’t be able to take anything at face value, not even the most innocuous looking expression. Getters and setters are tricky enough as it is.
I wholeheartedly agree with this assessment.
Metaprogramming is incredibly powerful, and has its place, but it generally comes down to 'defining your own language'. It cannot merely be Javascript any more: one must learn the idioms of your own personal language which Just Happens to run as Javascript.
As a junior dev I used to make maximal use of every language and platform capability I could find. These days, I'd rather just stick to the idioms and conventions of the parent language, because I want neither to maintain a compiler nor teach our junior devs how to use My Custom Language. It's enough to know that those capabilities are there without playing with them constantly.
Boring is simple and maintainable. Best to confine the interesting to the 'what' (which it'll infest anyway) rather than the 'how' (which can at least be kept under control).
Metaprogramming is incredibly powerful, and has its place, but it generally comes down to 'defining your own language'. It cannot merely be Javascript any more: one must learn the idioms of your own personal language which Just Happens to run as Javascript.
As a junior dev I used to make maximal use of every language and platform capability I could find. These days, I'd rather just stick to the idioms and conventions of the parent language, because I want neither to maintain a compiler nor teach our junior devs how to use My Custom Language. It's enough to know that those capabilities are there without playing with them constantly.
Boring is simple and maintainable. Best to confine the interesting to the 'what' (which it'll infest anyway) rather than the 'how' (which can at least be kept under control).
Of course it does. The point was to demonstrate the technique and give a trivial enough example to grasp what's going on. You wouldn't use it for this for the reasons you give.
The availability and accessibility of metaprogramming in Ruby is probably the closest one can get to an objective argument that it's a better language than other dynamic languages.
The superpowers metaprogramming gives you are abstraction power tools. If a particular abstraction doesn't really work for a given application, you can use metaprogramming to interface around the abstraction. You usually don't need it, but when you do, the alternative is a painful refactoring mess.
They allow you to quickly sketch out prototype code to get a trivial implementation working, and when you're done you can stuff all the code into a method on a module and call it a day. Tomorrow, when you need to instrument it somehow, pull out variables to be passed in, given default values, extend it to work with new formats, you don't have to worry about the language fighting you, and if it does, you can whip out metaprogramming to obliterate the offending semantics, all the way to down to BasicObject and eval if need be.
Metaprogramming allows you to mold abstractions like they were clay. In Javascript you just have to suffer if the language fights you. If you see code you can't understand, there's not much that can be done other than to beat your head against it until it works.
You don't have to understand how Ruby code works in order to work easily and effectively with it. All you simply have to know is that it works, then to extend you you can simply build an abstraction around it, even if it comes down to concatenating Ruby strings and then running eval on them. You can simply solve the problem quickly and dirtily, and then when time isn't a pressure, you can return to the code and clean up your mess.
Things you simply cannot do in other dynamic programming languages, to say nothing of statically-typed ones. The sheer speed at which you can work can be scary.
This might seem like fanboy worship but I seriously think that the only language that exists today that we'll still be using in a thousand years is a form of Ruby.
The availability and accessibility of metaprogramming in Ruby is probably the closest one can get to an objective argument that it's a better language than other dynamic languages.
The superpowers metaprogramming gives you are abstraction power tools. If a particular abstraction doesn't really work for a given application, you can use metaprogramming to interface around the abstraction. You usually don't need it, but when you do, the alternative is a painful refactoring mess.
They allow you to quickly sketch out prototype code to get a trivial implementation working, and when you're done you can stuff all the code into a method on a module and call it a day. Tomorrow, when you need to instrument it somehow, pull out variables to be passed in, given default values, extend it to work with new formats, you don't have to worry about the language fighting you, and if it does, you can whip out metaprogramming to obliterate the offending semantics, all the way to down to BasicObject and eval if need be.
Metaprogramming allows you to mold abstractions like they were clay. In Javascript you just have to suffer if the language fights you. If you see code you can't understand, there's not much that can be done other than to beat your head against it until it works.
You don't have to understand how Ruby code works in order to work easily and effectively with it. All you simply have to know is that it works, then to extend you you can simply build an abstraction around it, even if it comes down to concatenating Ruby strings and then running eval on them. You can simply solve the problem quickly and dirtily, and then when time isn't a pressure, you can return to the code and clean up your mess.
Things you simply cannot do in other dynamic programming languages, to say nothing of statically-typed ones. The sheer speed at which you can work can be scary.
This might seem like fanboy worship but I seriously think that the only language that exists today that we'll still be using in a thousand years is a form of Ruby.
Lisp has these advantages too, to an infinite degree. It's a wonderful teaching language for this reason: you can literally express any concept any way, introspect upon it, and execute it however.
The downside is that you end up building a custom language in the process... It can make one developer incredibly productive, to an almost superhuman degree, and maybe you can scale to a small team, but inducting others into the priesthood can take a very long time. Code needs a limited number of conventions, patterns and cliches so newcomers can rapidly train their mental dictionaries with the common stuff, so they can see through it to what it actually does.
> You don't have to understand how Ruby code works in order to work easily and effectively with it. All you simply have to know is that it works, then to extend you you can simply build an abstraction around it, even if it comes down to concatenating Ruby strings and then running eval on them. You can simply solve the problem quickly and dirtily, and then when time isn't a pressure, you can return to the code and clean up your mess.
Building an abstraction around something requires that you understand it, or at least how it interacts with the wider world, if that abstraction is not to become the next serious problem itself. (Most docs, even the 'thorough' ones, cover inputs and outputs only: environmental and causal dependencies are rarely properly documented and can really hurt.)
Otherwise it's called 'wrapping it in a bin-bag' and should be labelled with 'don't use this anywhere else unless you're prepared to open it and breathe deeply.'
It's generally been my experience that if you don't know how it works then it doesn't work, and rarely does anyone sufficiently time-constrained to do it 'dirtily' the first time ever have the time to come back and do it properly later: if it has to be quick and dirty, it should at least be transparent enough that someone else can deal with it.
[edit] Also, in a thousand years, Lisp will still exist, because absolutely everything else including Ruby is a mere subset. No one will use it though.
The downside is that you end up building a custom language in the process... It can make one developer incredibly productive, to an almost superhuman degree, and maybe you can scale to a small team, but inducting others into the priesthood can take a very long time. Code needs a limited number of conventions, patterns and cliches so newcomers can rapidly train their mental dictionaries with the common stuff, so they can see through it to what it actually does.
> You don't have to understand how Ruby code works in order to work easily and effectively with it. All you simply have to know is that it works, then to extend you you can simply build an abstraction around it, even if it comes down to concatenating Ruby strings and then running eval on them. You can simply solve the problem quickly and dirtily, and then when time isn't a pressure, you can return to the code and clean up your mess.
Building an abstraction around something requires that you understand it, or at least how it interacts with the wider world, if that abstraction is not to become the next serious problem itself. (Most docs, even the 'thorough' ones, cover inputs and outputs only: environmental and causal dependencies are rarely properly documented and can really hurt.)
Otherwise it's called 'wrapping it in a bin-bag' and should be labelled with 'don't use this anywhere else unless you're prepared to open it and breathe deeply.'
It's generally been my experience that if you don't know how it works then it doesn't work, and rarely does anyone sufficiently time-constrained to do it 'dirtily' the first time ever have the time to come back and do it properly later: if it has to be quick and dirty, it should at least be transparent enough that someone else can deal with it.
[edit] Also, in a thousand years, Lisp will still exist, because absolutely everything else including Ruby is a mere subset. No one will use it though.
> The downside is that you end up building a custom language in the process
You'll do it in many complex programs in many programming languages. Creating the necessary infrastructure in a programming language is called 'Greenspunning'.
If we for example need to work with 'state machines', I would implement in Lisp the machinery for it and give it a nicer source code user interface via some macro. The effect could be that the particular state machine definition sees a reduction in code size by, say, factor ten. Now productivity may depend on the size of the code -> productivity goes up. Readability and maintainability of the code goes up, too.
Now, if we program the same thing in Java or C++ - would we want to manually code the state machine without linguistic abstraction, where we would like to get shorter source code? Probably not. There are now several ways to work around this. One would be to have an XML scheme describing state machines and a translator to Java code. Or defining a custom external DSL for it. This approach is quite popular. Now you need to learn Java and your custom configuration language, plus its implementation.
Lisp just has a different solution for it: developers tend to implement kind of an internal macro-based DSL for these problems. The solution for that may look different from what Java or C++ uses - but they also need to abstract the code. If the code does not -> then you have here the reason why Lisp teams are smaller -> they are more productive. If a C++ program gets large enough, team size may go up ten times... I once heard an example for that from a Lucent manager who worked with a team of hundred people using Lisp.
You'll do it in many complex programs in many programming languages. Creating the necessary infrastructure in a programming language is called 'Greenspunning'.
If we for example need to work with 'state machines', I would implement in Lisp the machinery for it and give it a nicer source code user interface via some macro. The effect could be that the particular state machine definition sees a reduction in code size by, say, factor ten. Now productivity may depend on the size of the code -> productivity goes up. Readability and maintainability of the code goes up, too.
Now, if we program the same thing in Java or C++ - would we want to manually code the state machine without linguistic abstraction, where we would like to get shorter source code? Probably not. There are now several ways to work around this. One would be to have an XML scheme describing state machines and a translator to Java code. Or defining a custom external DSL for it. This approach is quite popular. Now you need to learn Java and your custom configuration language, plus its implementation.
Lisp just has a different solution for it: developers tend to implement kind of an internal macro-based DSL for these problems. The solution for that may look different from what Java or C++ uses - but they also need to abstract the code. If the code does not -> then you have here the reason why Lisp teams are smaller -> they are more productive. If a C++ program gets large enough, team size may go up ten times... I once heard an example for that from a Lucent manager who worked with a team of hundred people using Lisp.
Good point, but creating infrastructure doesn't necessarily mean creating idioms on the compiler level: neither DSLs nor macros. I would reject both without solid evidence that a good method and object API in the host language couldn't solve the problem first. My approach in C++, Java, C#, JS, etc would be to provide classes and methods which assist in building the state machine definition. If their existing compiler capabilities can be used to optimise it, so much the better, but I'd stick to the host language unless I had other reasons to delve into building my own language.
As a concrete example: some time ago we tried creating a domain-specific language to define some fairly hefty structures in our application. The DSL was more concise and was intended to provide a 'pit of success' for adding new such structures, by adding some extra verification and getting rid of boilerplate which might get miswritten or forgotten.
Some years later we finally found the time to throw out the DSL and replace it with straightforward (albeit slightly more verbose) compiled code with helper methods. Which, funnily enough, was a direct translation of what the DSL interpreter/compiler was doing anyway under the hood.
We threw out an extra build/start-up phase, a few thousand lines of pointless glue, and a massive debugging headache. We gained the strong typing which our platform provides by default.
The mistake we made was that the DSL was merely an abbreviation of code which was otherwise running in the exact same context as everything else. It was a syntax hack. It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely.
If you don't need to mess with syntax, you probably don't need metaprogramming. And syntax is not really the hard part of programming (if it's consistent), it should be just a minor and necessary hurdle before you get to the important things, so you probably don't need to mess with syntax either. Boilerplate and bloat can be solved with good API design, but only if the pointless repetition is really pointless repetition. If it's not easy to fix with a few helper functions, odds are it's not actually as common and repetitive as you'd like to believe.
Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project.
The caveat here of course is that, with Lisp, there's no difference between macros and API (the attempt to differentiate is itself meaningless). And that's fine, but it's not the case for other platforms which lack Lisp-style macros, and it doesn't help with accessibility. Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff.
(It was rather fun to abuse C# `this[]` getters to force assignment of a function result via the compiler, at one stage. While useful as a temporary refactoring tool, I'm glad that hack has finally left our codebase.)
(The dark side of 'no DSL' is the fluent interface. These are really hard to design well, and most are terrible. The examples always read nicely of course, because they're basically the design documents.)
(I'm also aware that there are valid use cases for building DSLs. I've just never encountered such a use case myself...)
As a concrete example: some time ago we tried creating a domain-specific language to define some fairly hefty structures in our application. The DSL was more concise and was intended to provide a 'pit of success' for adding new such structures, by adding some extra verification and getting rid of boilerplate which might get miswritten or forgotten.
Some years later we finally found the time to throw out the DSL and replace it with straightforward (albeit slightly more verbose) compiled code with helper methods. Which, funnily enough, was a direct translation of what the DSL interpreter/compiler was doing anyway under the hood.
We threw out an extra build/start-up phase, a few thousand lines of pointless glue, and a massive debugging headache. We gained the strong typing which our platform provides by default.
The mistake we made was that the DSL was merely an abbreviation of code which was otherwise running in the exact same context as everything else. It was a syntax hack. It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely.
If you don't need to mess with syntax, you probably don't need metaprogramming. And syntax is not really the hard part of programming (if it's consistent), it should be just a minor and necessary hurdle before you get to the important things, so you probably don't need to mess with syntax either. Boilerplate and bloat can be solved with good API design, but only if the pointless repetition is really pointless repetition. If it's not easy to fix with a few helper functions, odds are it's not actually as common and repetitive as you'd like to believe.
Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project.
The caveat here of course is that, with Lisp, there's no difference between macros and API (the attempt to differentiate is itself meaningless). And that's fine, but it's not the case for other platforms which lack Lisp-style macros, and it doesn't help with accessibility. Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff.
(It was rather fun to abuse C# `this[]` getters to force assignment of a function result via the compiler, at one stage. While useful as a temporary refactoring tool, I'm glad that hack has finally left our codebase.)
(The dark side of 'no DSL' is the fluent interface. These are really hard to design well, and most are terrible. The examples always read nicely of course, because they're basically the design documents.)
(I'm also aware that there are valid use cases for building DSLs. I've just never encountered such a use case myself...)
> provide classes and methods which assist in building the state machine definition
Now you shift the complexity to a language which is potentially bloated and less suitable to express domain level concepts in a concise way. We've see a lot of OO-architectures which try to recover flexibility by providing complex meta-level mechanisms. For a state-machine one would implement kind of an interpreter over an OO-data model - or a code generator from that oo-model -> greenspunning.
> It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely
For Lisp this would be natural. One can easily hide an implementation behind a domain-level descriptive representation of the problem. The distance between both is very small. It's actually what I would try to approach: working on a descriptive level with domain concepts and hiding the implementation.
> Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project.
That depends on the 'single project' and its size. Larger applications usually contain a multitude of such tailored notations and machineries to implement them.
> The caveat here of course is that, with Lisp, there's no difference between macros and API
The API consists of exported and documented macros.
> Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff.
Sure. If you look at the Common Lisp Object System, it was originally an extension to Common Lisp to implement an object system with classes, functions, methods, inheritance, etc.
The original implementation was in several layers:
The lowest level was a layer of objects. CLOS implemented in terms of itself. Like the definition of a class for classes. It's a bit unusual, since it is an implementation in itself.
The next layer was a layer of extensible functions and classes. Like creating classes by calling functions.
The developer layer is a level of macros where, for example, classes are specified via macros. The macros assemble the necessary language constructs - like how to descriptively define classes in a convenient notation.
The layers are documented and the lower layers are collections of protocols over classes and functions.
This layered language approach in Lisp is described in the book 'The Art of the Meta-Object Protocol', short AMOP.
https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Prot...
Now you shift the complexity to a language which is potentially bloated and less suitable to express domain level concepts in a concise way. We've see a lot of OO-architectures which try to recover flexibility by providing complex meta-level mechanisms. For a state-machine one would implement kind of an interpreter over an OO-data model - or a code generator from that oo-model -> greenspunning.
> It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely
For Lisp this would be natural. One can easily hide an implementation behind a domain-level descriptive representation of the problem. The distance between both is very small. It's actually what I would try to approach: working on a descriptive level with domain concepts and hiding the implementation.
> Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project.
That depends on the 'single project' and its size. Larger applications usually contain a multitude of such tailored notations and machineries to implement them.
> The caveat here of course is that, with Lisp, there's no difference between macros and API
The API consists of exported and documented macros.
> Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff.
Sure. If you look at the Common Lisp Object System, it was originally an extension to Common Lisp to implement an object system with classes, functions, methods, inheritance, etc.
The original implementation was in several layers:
The lowest level was a layer of objects. CLOS implemented in terms of itself. Like the definition of a class for classes. It's a bit unusual, since it is an implementation in itself.
The next layer was a layer of extensible functions and classes. Like creating classes by calling functions.
The developer layer is a level of macros where, for example, classes are specified via macros. The macros assemble the necessary language constructs - like how to descriptively define classes in a convenient notation.
The layers are documented and the lower layers are collections of protocols over classes and functions.
This layered language approach in Lisp is described in the book 'The Art of the Meta-Object Protocol', short AMOP.
https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Prot...
One of the unexpected behaviours was that if you proxy an object you cannot call it as a function even if you define `apply` handler. This made proxying async functions impossible because they always return Promise<T> even if T is another function, unless you wrap promise in a dummy function before proxying.
https://github.com/kozhevnikov/proxymise/blob/master/index.j...
https://github.com/kozhevnikov/proxymise/blob/master/index.j...
That is really interesting, I'm building something with Proxies for any Type and async function proxified worked for me. Mind if I hit you up for a chat?
Sure. I mean if you have an async function that returns another function (so it actually returns a promise of a function because it's async) proxy of the original function will call apply handler when invoked, but proxy of the return value will not because it's actually not a function but an object (a promise of a function) and as it turned out proxied objects do not call apply handler when invoked (even if it's defined) but throw an exception.
const handler = { apply: () => { console.log('ok') } };
const foo = new Proxy(() => {}, handler);
const bar = new Proxy({}, handler);
> foo()
< ok
> bar()
< VM313:1 Uncaught TypeError: bar is not a function
at <anonymous>:1:1One fun use case is that you can use proxies to detect accesses to nonexistent properties of JS objects, and you can then throw or log a warning rather than having them just return undefined. Unclear if you'd want to do that everywhere (due to perf reasons), but might might be a good thing in development/tests.
You can also use proxies to aid in code migrations. For example, let's say you want to migrate your frontend code from `user.id` to `user.handle` and eventually stop returning `id` from the backend. If you wrap server responses in a proxy, then you can warn in production if any code ever accesses `.id`. That's a lot more comforting than trying to search the code, and even in a 100% TypeScript codebase you still might have some dynamic accesses that the proxy would catch.
(I haven't actually done either of the two above approaches in a real-world scenario, but they've always seemed appealing, and I'm curious of others have real-world experience with them.)
You can also use proxies to aid in code migrations. For example, let's say you want to migrate your frontend code from `user.id` to `user.handle` and eventually stop returning `id` from the backend. If you wrap server responses in a proxy, then you can warn in production if any code ever accesses `.id`. That's a lot more comforting than trying to search the code, and even in a 100% TypeScript codebase you still might have some dynamic accesses that the proxy would catch.
(I haven't actually done either of the two above approaches in a real-world scenario, but they've always seemed appealing, and I'm curious of others have real-world experience with them.)
Yet another way JavaScript has become more like Perl5. It now features `tie`. https://perldoc.perl.org/functions/tie.html
I’ve developed a proof-of-concept library that uses them to provide optionals support (no more TypeErrors for writing things like obj.property.property2.property3 when property doesn’t exist) here: https://github.com/jdelman/proxy-lens
(Btw, I know calling it a lens isn’t right.)