The Lindy Effect(arxiv.org)
arxiv.org
The Lindy Effect
https://arxiv.org/abs/2308.09045
87 comments
> As a result, I almost never buy (or often even read) current-day bestsellers.
I like this strategy because it maximizes the value you get out of old books while you minimize the value of new books. A phenomenal and important book could be published today but the optimal strategy is to die before the Lindy effect becomes apparent and let another generation enjoy its value. This makes sense because books have a finite value that is divisible by the number of readers they get.
I like this strategy because it maximizes the value you get out of old books while you minimize the value of new books. A phenomenal and important book could be published today but the optimal strategy is to die before the Lindy effect becomes apparent and let another generation enjoy its value. This makes sense because books have a finite value that is divisible by the number of readers they get.
Why do you think books have a finite value divisible by the number of readers they get? That sounds like a very odd claim to me.
Let's say there are two universes U1 and U2 and a book A which is identical in both. In U1, book A gets 100 readers, while the same book in U2 gets 1000 readers. I think most people would say that the book has the same value in both universes, independent of how many read it.
Maybe your thought process involves the physical condition of a book (there is more wear and tear as it passes through more readers), but I'd like to understand your point of view.
Let's say there are two universes U1 and U2 and a book A which is identical in both. In U1, book A gets 100 readers, while the same book in U2 gets 1000 readers. I think most people would say that the book has the same value in both universes, independent of how many read it.
Maybe your thought process involves the physical condition of a book (there is more wear and tear as it passes through more readers), but I'd like to understand your point of view.
You overfit for classics and skip on maybe very entertaining contemporary reads. Feels weird to me to hipster out on the "I build a library of classics only". With this approach you'd never read Dark Matter, which is one of my favorites in the past few years. I also don't care about rankings, top sellers etc. etc., but contemporary stuff is still worth the read. Feels very weird to me tbh.
You're not wrong, which is why for me longevity is only one of the factors that goes into my book select - but it is a big one.
The thing is, we have so much brilliant art and literature available that you need to filter it somehow. The likelihood that something released this year is better than something that has been heralded and great for 100 years is very, very low. Not impossible, mind you. I've definitely read modern books that have had a huge impact on me (including, ironically, Taleb, who introduced me to the Lindy effect), and art and music have a certain cultural context that can make modern expressions more relevant.
The thing is, we have so much brilliant art and literature available that you need to filter it somehow. The likelihood that something released this year is better than something that has been heralded and great for 100 years is very, very low. Not impossible, mind you. I've definitely read modern books that have had a huge impact on me (including, ironically, Taleb, who introduced me to the Lindy effect), and art and music have a certain cultural context that can make modern expressions more relevant.
I agree! There are so many classics that are just a pain in the ass to read (looking at you Faust) and just plain suck. But I do love to read some of them sometimes.
I just sell my books quite frequently and keep only the ones that resonated with me. I'll also buy books after having read them as an ebook
I just sell my books quite frequently and keep only the ones that resonated with me. I'll also buy books after having read them as an ebook
Also, we are too kind to some classics out of nostalgia perhaps. I've been greatly disappointed by top rated classic spy and sci-fi novels. The kind of storytelling people got away with 50 years ago really doesn't work anymore today. Think of bad pacing, characters without depth, lazy stereotyping, deus ex machinas.
So on the one hand recency bias results in us giving undue attention to new books. On the other hand we are too forgiving to classics. There is no lesson here. Finding the right books to read is hard and you can't trust reviews.
So on the one hand recency bias results in us giving undue attention to new books. On the other hand we are too forgiving to classics. There is no lesson here. Finding the right books to read is hard and you can't trust reviews.
"The kind of storytelling people got away with 50 years ago really doesn't work anymore today."
Yeah, I've struggled with this a lot -- e.g. still having trouble reading the works of Dostoevsky because of those endless walking-in-the-park-and-philosophising dialogues. These seem alien in my time and culture.
I suppose you have to sense the characteristics of that 50-year-old language -- and see the universal, timeless ideas behind that language. Language is a tool, a form. It does change quite a bit in 50 (or 100) years, whereas the general ideas change much less IMO. To a large degree, this is a "seeing forest for the trees" thing.
Yeah, I've struggled with this a lot -- e.g. still having trouble reading the works of Dostoevsky because of those endless walking-in-the-park-and-philosophising dialogues. These seem alien in my time and culture.
I suppose you have to sense the characteristics of that 50-year-old language -- and see the universal, timeless ideas behind that language. Language is a tool, a form. It does change quite a bit in 50 (or 100) years, whereas the general ideas change much less IMO. To a large degree, this is a "seeing forest for the trees" thing.
Ohhh how I can relate. I'm reading Walden as a non-native speaker and 170 years ago things were quite different. Kindle has the nice functionality that tries to explain terms. What it does in practice tho is that it explains things I know and doesn't explain wildly exotic terms ...
Yeah I tried reading Solaris and was laughing at how bad it was. Felt the same way about nueromancer. Had a good run of trying out sci fi classics and being dissapointed. Philip k dicks stuff was solid tho. Ubiq is the shit
I'm also having a hard time with neuromancer, but I'm afraid that Gibson did this on purpose
Absolutely! Book discovery IS hard and it's sad that goodreads is still the best way for me. Recommendations suck on all platforms and visiting book shops has a recency bias and I'm going out with a whole in my pocket
The fact that you mention Dark Matter (no hate, taste is subjective etc.) as potentially rivaling actual classics is for me just further proof that fighting recency bias and hype is important.
But yes, of course some modern books will end up standing the test of time as well. But it's hard to judge that a priori without actually reading the book yourself.
You will spend a finite amount of time reading in your life. If you read Dark Matter now, that could for example mean that you will never end up reading Dune, or The Count of Monte-Cristo. If you're gonna sit down and read, an older book is a much safer bet if you want to maximize the value of your limited time.
Still, modern books can sometimes offer other qualities that simply don't exist in older books. Sci-Fi is a great example of that, where scienctific developments affect the writing deeply. So (unfortunately) in certain cases you might have to gamble on a newer book. But I try to avoid that.
But yes, of course some modern books will end up standing the test of time as well. But it's hard to judge that a priori without actually reading the book yourself.
You will spend a finite amount of time reading in your life. If you read Dark Matter now, that could for example mean that you will never end up reading Dune, or The Count of Monte-Cristo. If you're gonna sit down and read, an older book is a much safer bet if you want to maximize the value of your limited time.
Still, modern books can sometimes offer other qualities that simply don't exist in older books. Sci-Fi is a great example of that, where scienctific developments affect the writing deeply. So (unfortunately) in certain cases you might have to gamble on a newer book. But I try to avoid that.
It's not about dark matter rivaling "classics" (which we established are shit enough most of the times anyway). And if you're US American, your classics are most probably not my classics.
> an older book is a much safer bet if you want to maximize the value of your limited time
I 100% disagree with that. Value is highly subjective. I've read a lot of the books that are considered as "the most important books from the western hemisphere" and let me tell you, most of it is garbage. It's just a classic because back in the day it was novel and exciting, nothing else like it. You can have a good time with both, modern and old books. It's about your selection process, not the age of the book.
Reading has multiple purposes. Entertainment and education, most importantly. So I read books depending on the goal I want to achieve.
PS: if you read the commercial garbage from reading lists and what's new, you'll have a bad time, granted. As everybody can write a book nowadays, quality is harder to find tho.
> an older book is a much safer bet if you want to maximize the value of your limited time
I 100% disagree with that. Value is highly subjective. I've read a lot of the books that are considered as "the most important books from the western hemisphere" and let me tell you, most of it is garbage. It's just a classic because back in the day it was novel and exciting, nothing else like it. You can have a good time with both, modern and old books. It's about your selection process, not the age of the book.
Reading has multiple purposes. Entertainment and education, most importantly. So I read books depending on the goal I want to achieve.
PS: if you read the commercial garbage from reading lists and what's new, you'll have a bad time, granted. As everybody can write a book nowadays, quality is harder to find tho.
Not considering myself a hipster. I'm closely following new releases on several fields (including experimental fiction and contemporary philosophy), but I like to wait -- typically some years -- before I really read them.
Fair enough
Same with films and videogames. With books and films it's probably more the current Zeitgeist that doesn't agree with me, so if something's hyped up I tend to think it's probably due to the current social activism (which I believe won't age well) permeating everything. There is more than enough media from the past hundred years that has stood the test of time.
> so my children would have a carefully-filtered collection of "classics" from many eras by the time they reach young adulthood.
I have that. My three adult children read none of them.
I have that. My three adult children read none of them.
adamnemecek(4)
There's an implicit value judgment there though: would you stick a book that exhibited the Lindy effect even if you fundamentally disapproved of it.
For example: the King James English Bible is surely very high on the Lindyness scale, massively influential, and on that basis deserves a place on every educated reader's library: but its also the source of bigotry, suffering and death for thousands of people. Do you stock it, or no?
For example: the King James English Bible is surely very high on the Lindyness scale, massively influential, and on that basis deserves a place on every educated reader's library: but its also the source of bigotry, suffering and death for thousands of people. Do you stock it, or no?
The bible is a hugely influential book. It is probably the most referenced book in western literature, art, and history.
Reading a book is not the same as agreeing with its premises.
One must also judge a book by their historical context.
Reading a book is not the same as agreeing with its premises.
One must also judge a book by their historical context.
My AP Lit teacher in high school encouraged us to read the bible - not for any type of religious reason - but because it is so frequently referenced by western literature, especially in older texts. If you don't pick up on the biblical allegories when reading something like Shakespeare then you'll miss critical details.
Every page of the KJB seems to have a phrase that has entered the language (e.g. give up the ghost) - though I can't be sure that those phrases weren't already popular and merely recorded.
If they were popular before, chanches are they were popular because of influence from older versions of the Bible.
Umm, no. There is a huge selection bias at work here. Most people, when they read the bible, read Genesis and Exodus, the Gospels, Psalms, Ecclesiastes and a couple of other books. It is true that lots of phrases, images, stories, quotations and so on from those heavy-hitting parts of the Bible have made their way into the general culture.
The vast majority of the Bible is not at all influential on either our religious understanding or on general culture. Partly because most of it just isn't that commonly read. The popular parts are read thousands of times more commonly than the least popular parts, and many of the readers of the least popular parts (in our era) are people who have a deep religious commitment to reading the Bible, but who aren't necessarily prolific writers or influential thinkers.
The other fact at play here is that most of the unread parts are unread for a reason. There is a lot of material which is, of course, of historical or theological interest to experts, and which had undeniable spiritual significance to many people who lived long ago, but which is very hard for most modern readers to extract any real meaning or value from.
Some examples - each of these is taken not from the really obscure books, but from 'second-tier books', in order to demonstrate how quickly the long tail of Bible literature descends to filler.
The book of Numbers (part of the Pentateuch/Torah) mainly consists of pages and pages listing how many animals of different types should be sacrificed in various ways for different festivals. This is extremely repetitive stuff, pretty much beyond satire. There is no justification given for any of this doctrine, and once you have extracted a minimal amount of historical content ("it seems like cattle were more valuable than sheep!") it's on a par with reading the telephone book.
The book of Proverbs sounds fun, but do you know any of the proverbs it contains? Probably not many, because they consist of dozens of very small variations on the same message to embrace wisdom, as though there were some doubt about the advantages of wisdom. Sample: "Blessings are on the head of the righteous, but the mouth of the wicked conceals violence." The catchy phrasing and apt metaphors which you would hope for simply aren't here (barring a very small number, which stand out as being slightly less banal: "iron sharpens iron").
A couple of the Epistles of St Paul are deeply uplifting and moving pronouncements of the essence of belief in Christ. The rest (circa 20 books) are probably fascinating political history, if you have the background knowledge about the different early Christian thinkers and communities and the evolution of their doctrine. If you don't, they are hard to make much sense of - for example, thirty pages of careful analytic debate about circumcision, responding to someone else's letter which itself breaks down a third person's arguments, neither of which we have available to us.
The awkward truth about the notion of the 'Bible as literature' or the supposed cultural importance of the Bible, is that in modern secular times, it is a myth passed on from otherwise educated people who have read less than 5% of the Bible, to other educated people who are going to read even less of it.
The vast majority of the Bible is not at all influential on either our religious understanding or on general culture. Partly because most of it just isn't that commonly read. The popular parts are read thousands of times more commonly than the least popular parts, and many of the readers of the least popular parts (in our era) are people who have a deep religious commitment to reading the Bible, but who aren't necessarily prolific writers or influential thinkers.
The other fact at play here is that most of the unread parts are unread for a reason. There is a lot of material which is, of course, of historical or theological interest to experts, and which had undeniable spiritual significance to many people who lived long ago, but which is very hard for most modern readers to extract any real meaning or value from.
Some examples - each of these is taken not from the really obscure books, but from 'second-tier books', in order to demonstrate how quickly the long tail of Bible literature descends to filler.
The book of Numbers (part of the Pentateuch/Torah) mainly consists of pages and pages listing how many animals of different types should be sacrificed in various ways for different festivals. This is extremely repetitive stuff, pretty much beyond satire. There is no justification given for any of this doctrine, and once you have extracted a minimal amount of historical content ("it seems like cattle were more valuable than sheep!") it's on a par with reading the telephone book.
The book of Proverbs sounds fun, but do you know any of the proverbs it contains? Probably not many, because they consist of dozens of very small variations on the same message to embrace wisdom, as though there were some doubt about the advantages of wisdom. Sample: "Blessings are on the head of the righteous, but the mouth of the wicked conceals violence." The catchy phrasing and apt metaphors which you would hope for simply aren't here (barring a very small number, which stand out as being slightly less banal: "iron sharpens iron").
A couple of the Epistles of St Paul are deeply uplifting and moving pronouncements of the essence of belief in Christ. The rest (circa 20 books) are probably fascinating political history, if you have the background knowledge about the different early Christian thinkers and communities and the evolution of their doctrine. If you don't, they are hard to make much sense of - for example, thirty pages of careful analytic debate about circumcision, responding to someone else's letter which itself breaks down a third person's arguments, neither of which we have available to us.
The awkward truth about the notion of the 'Bible as literature' or the supposed cultural importance of the Bible, is that in modern secular times, it is a myth passed on from otherwise educated people who have read less than 5% of the Bible, to other educated people who are going to read even less of it.
You clearly are more informed than me, but my experience was from randomly reading pages. Maybe I got lucky. Though I recall one page with long lists of ancestry, x begat y - similar to a telephone book as you mention.
Abraham was the father of Isaac, Isaac the father of Jacob, Jacob the father of Judah and his brothers, Judah the father of Perez and Zerah, whose mother was Tamar, Perez the father of Hezron, Hezron the father of Ram, Ram the father of Amminadab, Amminadab the father of Nahshon, Nahshon the father of Salmon, [...]
You do not need the whole bible for that. The bible contain way more then cultural references and a lot of that more is mindnumbing boring.
You do not need the whole bible for that. The bible contain way more then cultural references and a lot of that more is mindnumbing boring.
That was the first thing I noticed when I attempted to read my KJV bible and also what eventually led me to stop reading it. The monotonous repetition of lineage. Perhaps I will give it another chance later in life.
The premise of the Bible is salvation from human’s fallen state through love, compassion and self sacrifice.
Many people can read the same book and get different things from it, which is an interesting point to consider wrt the lindy effect. Some books are read differently over time, or by different groups/nations. The lindy effect can include books that are read for different reasons; undoubtedly people still read mein kampf, but for wholly different reasons (one hopes!)
He who drinks the old wine is probably more interested in history than he who drinks the new.
Many people can read the same book and get different things from it, which is an interesting point to consider wrt the lindy effect. Some books are read differently over time, or by different groups/nations. The lindy effect can include books that are read for different reasons; undoubtedly people still read mein kampf, but for wholly different reasons (one hopes!)
He who drinks the old wine is probably more interested in history than he who drinks the new.
TBF that's less because of the actual content of the bible, but what people in various eras read into it (except for some of the Old Testament "eye for an eye" stuff of course).
PS: ...and even though I'm agnostic, of course I have the bible in my book shelf and occasionally read it, if only for the entertainment value (not the King James version though, but the Luther version).
PS: ...and even though I'm agnostic, of course I have the bible in my book shelf and occasionally read it, if only for the entertainment value (not the King James version though, but the Luther version).
I definitely would stock the Bible, yes -- with the argument that Western societies and cultures are riddled with biblical metaphors and everyday customs, so it does make sense for the kids to be familiar with them.
Not sure about stocking e.g. Mein Kampf, though. This one I would probably pass, due to disgust towards what the Nazis did. From another angle, I might be "brainwashed" here -- by the cultural meme that this is a Dangerous Book. I don't really know. In the end, it is still just a book. Can a book be dangerous in itself? Not sure that possessing it would turn somebody into a Nazi; a youngster's appreciation of those ideas is surely more complicated a mechanism than just reading a book.
Tangentially related: in my (post-soviet) country, there has been discussion whether former Soviet military buildings are a part of cultural heritage, or should they all be demolished. As a regular person, I'm definitely siding with keeping some of those buildings -- the Soviet occupation was a sad, devastating part of our past, and younger generations should have physical examples of what it was like. Whether those buildings will be used as some kind of cult objects etc -- this is a matter of values education at school or at home.
So I fully agree with @j7ake's comment in this thread that reading (or -- my addition -- possessing) a book is not the same as agreeing with its premises. Hiding the past won't work, I think.
Great angle @kjellsbells, though. Interesting to think about. Thanks for bringing this up.
Not sure about stocking e.g. Mein Kampf, though. This one I would probably pass, due to disgust towards what the Nazis did. From another angle, I might be "brainwashed" here -- by the cultural meme that this is a Dangerous Book. I don't really know. In the end, it is still just a book. Can a book be dangerous in itself? Not sure that possessing it would turn somebody into a Nazi; a youngster's appreciation of those ideas is surely more complicated a mechanism than just reading a book.
Tangentially related: in my (post-soviet) country, there has been discussion whether former Soviet military buildings are a part of cultural heritage, or should they all be demolished. As a regular person, I'm definitely siding with keeping some of those buildings -- the Soviet occupation was a sad, devastating part of our past, and younger generations should have physical examples of what it was like. Whether those buildings will be used as some kind of cult objects etc -- this is a matter of values education at school or at home.
So I fully agree with @j7ake's comment in this thread that reading (or -- my addition -- possessing) a book is not the same as agreeing with its premises. Hiding the past won't work, I think.
Great angle @kjellsbells, though. Interesting to think about. Thanks for bringing this up.
[deleted]
Mein Kampf is worth at least a skim, because it demonstrates that Hitler was a human being.
History positions him as an inhuman monster, and rightly so metaphorically. We are conditioned to think that he was uniquely evil in an otherworldly sense, totally unlike us, and someone like him could never arise again because we don’t have monsters these days.
But he wasn’t a unique demon. He was a human, with human emotions and motivations and failings. It’s important to recognize him (and all the rest of the Nazis) as human, because the humans of today absolutely do have the capacity to become him again.
History positions him as an inhuman monster, and rightly so metaphorically. We are conditioned to think that he was uniquely evil in an otherworldly sense, totally unlike us, and someone like him could never arise again because we don’t have monsters these days.
But he wasn’t a unique demon. He was a human, with human emotions and motivations and failings. It’s important to recognize him (and all the rest of the Nazis) as human, because the humans of today absolutely do have the capacity to become him again.
No and this is also a thing only people do not know what is in it say. Or people who want to be edgy say. And I am saying that as someone who attempted to read it, so have passing idea about content.
If you want to know who Hitler was as a human, read either "Hitler, by Kershaw Ian" or "Third Reich Trilogy by Richard Evans".
If you want someone to realize Hitler was not unique, make them read about other genocides, about Italy at the same time, about Stalin, about Japan, about slavery, about genocide of native americans and so on.
Also, I would highly NOT recommend it for someone who does not know actual German history, because that book contains ... well quite a few untrue claims. If you are in general interested in that period and read a lot, nothing wrong with reading it. But if you are not and this is the one historical book you decided to pick and recommend over all the others, then I have hard time to believe you are neutral in your pick.
If you want to know who Hitler was as a human, read either "Hitler, by Kershaw Ian" or "Third Reich Trilogy by Richard Evans".
If you want someone to realize Hitler was not unique, make them read about other genocides, about Italy at the same time, about Stalin, about Japan, about slavery, about genocide of native americans and so on.
Also, I would highly NOT recommend it for someone who does not know actual German history, because that book contains ... well quite a few untrue claims. If you are in general interested in that period and read a lot, nothing wrong with reading it. But if you are not and this is the one historical book you decided to pick and recommend over all the others, then I have hard time to believe you are neutral in your pick.
"Mein Kampf is worth at least a skim, because it demonstrates that Hitler was a human being."
Excellent phrasing. I think you would love "The Kindly Ones" by Jonathan Littell [1]. What an incredible book -- oh, and, being released in 2009, it also fits my interpretation of "Lindy".
1: https://en.wikipedia.org/wiki/The_Kindly_Ones_(Littell_novel...
Excellent phrasing. I think you would love "The Kindly Ones" by Jonathan Littell [1]. What an incredible book -- oh, and, being released in 2009, it also fits my interpretation of "Lindy".
1: https://en.wikipedia.org/wiki/The_Kindly_Ones_(Littell_novel...
Really? The bible is the source of bigotry and death for thousands? A vit dramatic no? The bible does not implore anyone to violence. But it is a convenient scapegoat for those with ill will
[deleted]
Yes definitely. Currently in four languages, for the record.
Yes. I'd stock Koran too. Satanic Verses by Rushdie would also be included. If I can have access to Mein Kampf, to the library it goes.
Unless you are specifically interested in the history of the response to the Satanic Verses and what it means about world society, I would recommend 'Midnight's Children' as a better book, and a better example of what is interesting about Salman Rushdie's work.
> Mein Kampf
While the author was very influential, I'm not sure if the book itself relevant.
While the author was very influential, I'm not sure if the book itself relevant.
Historically speaking, it is very relevant. Because it clearly shows Hitler's intent for Germany, the war and the jews way before he got into power. Other than that, it is a crappy book, the content sucks and the writing is abysmally bad. The latter fact was also something people thought back then, and most people who had the book at home didn't necessarily read it.
Another reason the book is relevant, it clearly shows the power of propaganda and how books can be used to that end.
Another reason the book is relevant, it clearly shows the power of propaganda and how books can be used to that end.
> Because it clearly shows Hitler's intent for Germany, the war and the jews way before he got into power.
It's interesting from the perspective of historiography. But it's not relevant in the way the Bible, Quran or the Gita are.
> Other than that, it is a crappy book, the content sucks and the writing is abysmally bad. The latter fact was also something people thought back then, and most people who had the book at home didn't necessarily read it.
This was my point. And in the context of TFA, it's not Lindy.
> clearly shows the power of propaganda
It wasn't an effective instrument for propaganda. The book became popular because of Hitler, not the other way round.
It's interesting from the perspective of historiography. But it's not relevant in the way the Bible, Quran or the Gita are.
> Other than that, it is a crappy book, the content sucks and the writing is abysmally bad. The latter fact was also something people thought back then, and most people who had the book at home didn't necessarily read it.
This was my point. And in the context of TFA, it's not Lindy.
> clearly shows the power of propaganda
It wasn't an effective instrument for propaganda. The book became popular because of Hitler, not the other way round.
"Mein Kampf" is actually a good example of the "negative" Lindy effect because it was only relevant in a single country for a very short time, and even in Nazi Germany it was only popular (in the sense that there was a copy in every household) because it was given out as wedding gift by the state (pretty much nobody could be arsed to actually buy and read that drivel when it came out).
This is historical revisionism. Mein Kampf was a decently well selling book even before the Nazis came to power, with an estimated number of 241000 copies sold.
This is of course dwarfed by the number of copies distributed after the Nazis came to power.
This is of course dwarfed by the number of copies distributed after the Nazis came to power.
> its also the source of bigotry, suffering and death for thousands of people
But are you afraid reading it will turn you into a murderous bigot?
But are you afraid reading it will turn you into a murderous bigot?
What someone says is the justification for their actions, and what actually is the reason for their actions are often different things.
Here is one way, I use the Lindy effect as a mental model:
When picking a tech stack for a software investment and you want a reasonable expectation that it will still be actively maintained 10 years from now, you need to go back in time by 10 years. I’ve found this 10 years time horizon a useful one to work with, based on version histories of various bits of software I build upon. For example, for Python, this takes me back to version 3.4, before type decorators and many other things hit the scene that I disapprove of. So now, I test all my code with both version 3.4 and the newest version (3.11). I won’t use language features from 3.11 that weren’t already there in 3.4, and I won’t use code in 3.4 that breaks or throws deprecation warnings in 3.11. I also apply this test to dependencies, paying close attention to what happens if I try to get the newest version of some Python library running on version 3.4 of the interpreter or a 10-year-old version of the library on version 3.11 of the interpreter.
This means my code is engineered so that it could have been in continuous operation for the last 10 years while running continuous updates on what’s underneath it. And this gives me a reasonable expectation that my code will require only minimal code changes over the next 10 years to keep up with whatever might arise.
With the python interpreter itself, it’s remarkably easy to do that. I feel it doesn’t limit my coding in any meaningful way. With other bits of software, including many python libraries that one might depend on, it would be absolutely unworkable. In such a case, I take that as a clear signal not to use the dependency at all. This is a lot of work, but also a good forcing function that prevents me from becoming a dependency hog myself.
When picking a tech stack for a software investment and you want a reasonable expectation that it will still be actively maintained 10 years from now, you need to go back in time by 10 years. I’ve found this 10 years time horizon a useful one to work with, based on version histories of various bits of software I build upon. For example, for Python, this takes me back to version 3.4, before type decorators and many other things hit the scene that I disapprove of. So now, I test all my code with both version 3.4 and the newest version (3.11). I won’t use language features from 3.11 that weren’t already there in 3.4, and I won’t use code in 3.4 that breaks or throws deprecation warnings in 3.11. I also apply this test to dependencies, paying close attention to what happens if I try to get the newest version of some Python library running on version 3.4 of the interpreter or a 10-year-old version of the library on version 3.11 of the interpreter.
This means my code is engineered so that it could have been in continuous operation for the last 10 years while running continuous updates on what’s underneath it. And this gives me a reasonable expectation that my code will require only minimal code changes over the next 10 years to keep up with whatever might arise.
With the python interpreter itself, it’s remarkably easy to do that. I feel it doesn’t limit my coding in any meaningful way. With other bits of software, including many python libraries that one might depend on, it would be absolutely unworkable. In such a case, I take that as a clear signal not to use the dependency at all. This is a lot of work, but also a good forcing function that prevents me from becoming a dependency hog myself.
Sounds like there would be barely any up to date dependency that you could use, which in some cases could have known security issues, no?
To clarify: I still run the latest version of all the packages I depend on. For example, the newest version of SSL that you can still use with Python 3.4 has known issues. But I don't use that in production. I use Python 3.11 with an up-to-date SSL, but the code that I've written myself is code that I've also tested on Python 3.4 with outdated SSL. So the subset of features I use is a set of features considered by Python and SSL maintainers as so essential that they've maintained compatibility around them for ten years, and are therefore likely to maintain compatibility around for another ten years. -- Whether that's true no one can know, but applying the Lindy effect mental model would imply that this would be a useful way to think about it.
The packages I get to use after applying the above filter are a minority, but, surprisingly, it's not "barely any" either. If you decide to live like this, you'll have less eyecandy in your life and less capability to capitalize on all the latest hotness all the time. But you'll get the job done, pretty much regardless of what the job might be.
Your dependency hog colleagues will call you a dinosaur for your choice of tech stack. But when security vulnerabilities happen then, more often than not, there won't be any action needed on your part because, more often than not, you'll discover that you don't depend on the thing that has the vulnerability in the first place. You'll be home for dinner, while those same colleagues try to patch their dependencies, watch their software break, and try to work their way out of dependency hell (presumably by introducing more dependencies).
The packages I get to use after applying the above filter are a minority, but, surprisingly, it's not "barely any" either. If you decide to live like this, you'll have less eyecandy in your life and less capability to capitalize on all the latest hotness all the time. But you'll get the job done, pretty much regardless of what the job might be.
Your dependency hog colleagues will call you a dinosaur for your choice of tech stack. But when security vulnerabilities happen then, more often than not, there won't be any action needed on your part because, more often than not, you'll discover that you don't depend on the thing that has the vulnerability in the first place. You'll be home for dinner, while those same colleagues try to patch their dependencies, watch their software break, and try to work their way out of dependency hell (presumably by introducing more dependencies).
Ah, ok, that makes way more sense.
I really wish that we, software developers, apply Lindy rule while choosing library/framework/language/SaaS tool etc., Lot of developers underestimate the value of collective brain power that has gone into making a system mature, not to talk about community and ecosystem support. I was extremely hesitant to get onboard Golang bandwagon 5-7 years ago. I reluctantly said OK to try it out in a non critical service and all my concerns turned out to be true.
I also apply this rule while buying car and choosing a bank. It's been interesting to see all the fintechs who badmouthed incumbent banks now going through exactly same challenges that banks went through and addressed.
I also apply this rule while buying car and choosing a bank. It's been interesting to see all the fintechs who badmouthed incumbent banks now going through exactly same challenges that banks went through and addressed.
Care to elaborate on the Go issues you encountered? Nowadays Go is very well established, unless you specifically meant that the issues happened 5-7 years ago.
Of the top of my head:
1) Lack of libraries.
2) Little to non-existent tooling & observability support.
3) Lots of challenges around gRPC. That it needed a discovery service such as etcd was a big source of pain. I remember two or three massive downtimes where an adjacent team went through hell for a day just to get back a high throughput system back online. gRPC + etcd was very susceptible to thundering herd problem.
4) Lack of experienced developers who would guide others idiomatic way of doing things. And related, go routines need a different mindset and somewhat a grounds up re-learning. The code would routinely get into deadlock sort of scenarios.
I conceded that most of the above are not inherent to Go but to any new system/language. I'm sure they all have been addressed now as the language and ecosystem has matured. That was the point I was trying to make though; don't jump onto a new bandwagon. Of course you want to learn it on your own and experiment that's totally fine but to use it in production is fraught with risk IMO.
1) Lack of libraries.
2) Little to non-existent tooling & observability support.
3) Lots of challenges around gRPC. That it needed a discovery service such as etcd was a big source of pain. I remember two or three massive downtimes where an adjacent team went through hell for a day just to get back a high throughput system back online. gRPC + etcd was very susceptible to thundering herd problem.
4) Lack of experienced developers who would guide others idiomatic way of doing things. And related, go routines need a different mindset and somewhat a grounds up re-learning. The code would routinely get into deadlock sort of scenarios.
I conceded that most of the above are not inherent to Go but to any new system/language. I'm sure they all have been addressed now as the language and ecosystem has matured. That was the point I was trying to make though; don't jump onto a new bandwagon. Of course you want to learn it on your own and experiment that's totally fine but to use it in production is fraught with risk IMO.
> One implication is that even things which are becoming less robust over time can display the Lindy effect.
Interesting. And somewhat counterintuitive.
Interesting. And somewhat counterintuitive.
There's a subset of this I think about sometimes, not sure if it's actually a thing or been looked at/is valid.
Let's say I'm waiting for a package to arrive and it's late. As each minute passes, does the chance of it coming the very next minute increase or decrease?
One could say it increases because since it will eventually come and I have passed all the previous minutes they haven't come, then the remaining minutes until it comes comprise a smaller percentage of the all the remaining minutes, increasing the chance each minute.
However up to a certain point the chance may decrease, as in, if it is "late" in a substantial sense, whatever caused it being something that delayed it a significant amount of time increases as time elapses, as the chances of it being something small that caused the lateness decreases as each minute passes.
Let's say I'm waiting for a package to arrive and it's late. As each minute passes, does the chance of it coming the very next minute increase or decrease?
One could say it increases because since it will eventually come and I have passed all the previous minutes they haven't come, then the remaining minutes until it comes comprise a smaller percentage of the all the remaining minutes, increasing the chance each minute.
However up to a certain point the chance may decrease, as in, if it is "late" in a substantial sense, whatever caused it being something that delayed it a significant amount of time increases as time elapses, as the chances of it being something small that caused the lateness decreases as each minute passes.
And you can model this through two survival functions, one thin-tailed and one heavy-tailed. As time increases, whatever your initial probability of "package will be very late" it will be dwarfed by the posterior after evidence of a long waiting time.
It's the exact same technique used in https://two-wrongs.com/forecasting-covid-19-variants.html except applied to "packages that may be late" instead.
It's the exact same technique used in https://two-wrongs.com/forecasting-covid-19-variants.html except applied to "packages that may be late" instead.
> it will eventually come
It might also never come.
Sometimes packages just disappear in transit.
Maybe it lost its label.
Maybe it got broken.
Maybe someone stole it.
Etc.
My experience with packages goes like:
- Most often things I order in my own country or from a neighbouring country arrive exactly on the expected day, or at least the next business day after that. This makes sense because shorter travel has less opportunity for things to go wrong.
- Things from far away have their estimate set for like 60 days from now, because no one knows what boat it will end up etc. Sometimes these things show up quickly. Most of the time it takes a while, but still it arrives before the long estimate date. Sometimes they show up a bit late.
- If they are very delayed then in both cases it means the package is just gone and will never arrive. So I inform the platform that the package was lost in transit and they refund the purchase.
It might also never come.
Sometimes packages just disappear in transit.
Maybe it lost its label.
Maybe it got broken.
Maybe someone stole it.
Etc.
My experience with packages goes like:
- Most often things I order in my own country or from a neighbouring country arrive exactly on the expected day, or at least the next business day after that. This makes sense because shorter travel has less opportunity for things to go wrong.
- Things from far away have their estimate set for like 60 days from now, because no one knows what boat it will end up etc. Sometimes these things show up quickly. Most of the time it takes a while, but still it arrives before the long estimate date. Sometimes they show up a bit late.
- If they are very delayed then in both cases it means the package is just gone and will never arrive. So I inform the platform that the package was lost in transit and they refund the purchase.
This is a sort of transformation of the bathtub curve.
Replace it with waiting for a bus and you have a classic textbook example for teaching Poisson process!
https://en.wikipedia.org/wiki/Poisson_point_process
https://stats.stackexchange.com/questions/122722/please-expl...
https://en.wikipedia.org/wiki/Poisson_point_process
https://stats.stackexchange.com/questions/122722/please-expl...
It's specifically not a Poisson process, which assumes that the chance of a bus arriving at any given time is constant.
A bus arrival has some kind of multimodal distribution, where the bus either comes on time or very late, with some noise in each case.
A bus arrival has some kind of multimodal distribution, where the bus either comes on time or very late, with some noise in each case.
The link between the Poisson process and the distribution of bus arrival times, is firstly that it's a good idealized example for understanding and reasoning about Poisson and exponential distribution, and secondly that the Poisson process somehow seems to be an accurate representation of our mental model of how waiting for a bus works.
If buses are scheduled to come quite frequently (and don't disappear, so they do arrive with that frequency on average), but the variance in their arrival times is a lot bigger than the average gap between them, then the distribution is actually quite close to Poisson.
If buses are scheduled to come quite frequently (and don't disappear, so they do arrive with that frequency on average), but the variance in their arrival times is a lot bigger than the average gap between them, then the distribution is actually quite close to Poisson.
> So something whose lifespan is exponentially distributed (such as a radioactive particle) does not display the Lindy effect.
Huh? Is this true? If I had particles with unknown half-times, I think I would expect the older particles to stay around longer, if that makes sense. (It is possible that there is no 'I expect' in the probability theory sense, like in the two envelopes problem.)
Huh? Is this true? If I had particles with unknown half-times, I think I would expect the older particles to stay around longer, if that makes sense. (It is possible that there is no 'I expect' in the probability theory sense, like in the two envelopes problem.)
Clearly, my choice to program in Free Pascal/Lazarus, using the old GUI toolkits is vindicated! ;-)
In the world of machining, 1940s-1960s American cast iron machine tools (Mills, Lathes, etc) are the best available. Grind / Scrape the bearing surfaces true, update the control systems, and they'll be good to go for at least 2 more generations.
In the world of machining, 1940s-1960s American cast iron machine tools (Mills, Lathes, etc) are the best available. Grind / Scrape the bearing surfaces true, update the control systems, and they'll be good to go for at least 2 more generations.
It seems like the Lindy effect for a positive real number valued ramdom variable X, is the following identity
E[X | X >= a] = 2 * a
E[X | X >= a] = 2 * a
> E[X | X >= a] = 2 * a
Is it possible to derive the maximum entropy probability distribution subject to that constraint?
Is it possible to derive the maximum entropy probability distribution subject to that constraint?
Let f: R -> R be the probability density function of X. It fulfills int f(x) dx = 1 and we furthermore assume that f is differentiable.
We can write the identity as
We integrate from t to x
I haven't double checked this so maybe there is a mistake, but I'm quite convinced that this line of reasoning leads to a unique distribution for each t > 0 and impossibility when t = 0.
We can write the identity as
int_a^oo x f(x) dx / int_a^oo f(x) dx = 2 a.
int_a^oo x f(x) dx = 2 a int_a^oo f(x) dx.
- a f(a) = 2 int_a^oo f(x) dx + 2 a (- f(a)).
- f(a) - a f'(a) = - 2 f(a) - 2 f(a) - 2 a f'(a).
f'(a) = - 3 f(a) / a,
f'(a) / f(a) = - 3 / a.
(ln ○ f)'(a) = - 3 / a.
We integrate from t to x
int_t^x (ln ○ f)'(a) da = int_t^x - 3 / a da.
ln(f(x)) = ln(f(t)) - int_t^x 3 / a da
= ln(f(t)) - 3 ln(x) + 3 ln(t)
= ln(f(t)) - ln(x^3) + ln(t^3)
= ln(f(t) t^3 / x^3).
f(x) = f(t) t^3 / x^3.
1 = int f(x) dx
= int_t^oo f(t) t^3 / x^3 dx
= f(t) t^3 [- 1 / (2 oo^2) + 1 / (2 t^2)]
= f(t) t / 2
f(t) = 2 / t
f(x) = 2 t^2 / x^3.
I haven't double checked this so maybe there is a mistake, but I'm quite convinced that this line of reasoning leads to a unique distribution for each t > 0 and impossibility when t = 0.
It's more likely described by the non-linear portion corresponding to
the first stage of the Bathtub Curve [0] we use in engineering.
[0] https://en.wikipedia.org/wiki/Bathtub_curve
Most complex but faulted things fail very early.
[0] https://en.wikipedia.org/wiki/Bathtub_curve
Most complex but faulted things fail very early.
“Tried and true”
Sayings about how old things are surprisingly effective:
"Don't knock it till you try it" "Chesterton's fence" "Reinventing the wheel"
Any others?
"Don't knock it till you try it" "Chesterton's fence" "Reinventing the wheel"
Any others?
“if it ain’t broke, don’t fix it”
“better the devil you know than the devil you don’t”
“old is gold”
“stood the test of time”
“better the devil you know than the devil you don’t”
“old is gold”
“stood the test of time”
The opposite of Chesterton's fence is "Chesterton's facehugger". Sometimes, when you discover something bad, the best strategy is to immediately try to remove it as quickly as possible, before spending any energy trying to find out why it appeared.
Not sure it fits, but I like The Gods of Copy Book Headings
https://en.m.wikipedia.org/wiki/The_Gods_of_the_Copybook_Hea...
https://en.m.wikipedia.org/wiki/The_Gods_of_the_Copybook_Hea...
java will continue for 28 years; c much longer; and when both are long gone, lisp and fortran will still be here.
As a result, I almost never buy (or often even read) current-day bestsellers. Usually, one is not in a hurry with reading a book; also, new books can be really well marketed these days, which always makes me cautious. It seems to make more sense to read a book 5-10 years after it was first released.
The Lindy effect does seem to hold true when I think about e.g. SICP, K&R's C book -- or Kernighan et al's 1988 AWK book for which the authors release a second edition in October this year! [1]
Not to mention classic books in fiction or philosophy. Or cultural memes and ideas -- as a rough heuristic, the Lindy effect does often seem to hold true with these as well. It is a fun and extremely useful thinking tool for longer-term everyday prognostics.
1: https://www.awk.dev/