We are drowning in information while starving for wisdom (2021)(realizeengineering.blog)
realizeengineering.blog
We are drowning in information while starving for wisdom (2021)
https://realizeengineering.blog/2021/01/20/we-are-drowning-in-information-while-starving-for-wisdom/
212 comments
Wisdom can't be squeezed out of a rock. The experience of wisdom is individual, because wisdom is transformative. What's being transformed is not the knowledge but you, your consciousness. It's local in that sense. It's not something that can be packaged or productized.
Wisdom comes from an awareness of the greater whole. The insights do not come from analysis, but rather, synthesis. It engages with the intuition rather than the intellect.
Although it can seem mystical, I think there are authors who have been able to express ways to engage in wisdom even if they are not directly talking about it. For example, Christopher Alexander has some interesting things to say about wholes, centers, and unfolding:
Seeing Wholes - https://iamronen.com/blog/2018/01/14/christopher-alexander-s...
Centers - https://iamronen.com/blog/2018/01/08/christopher-alexander-c...
Unfolding - https://www.livingneighborhoods.org/ht-0/whatisanunfolding.h...
Wisdom comes from an awareness of the greater whole. The insights do not come from analysis, but rather, synthesis. It engages with the intuition rather than the intellect.
Although it can seem mystical, I think there are authors who have been able to express ways to engage in wisdom even if they are not directly talking about it. For example, Christopher Alexander has some interesting things to say about wholes, centers, and unfolding:
Seeing Wholes - https://iamronen.com/blog/2018/01/14/christopher-alexander-s...
Centers - https://iamronen.com/blog/2018/01/08/christopher-alexander-c...
Unfolding - https://www.livingneighborhoods.org/ht-0/whatisanunfolding.h...
I think what you are talking about are what a few hundred years ago you called prophets. Some people can pass on wisdom but it’s a very rare skill, and followers can often exhibit the cult like behaviour that’s so repulsive to others. Probably because they can’t pass on that wisdom to others.
Also, many so called prophets can easily exploit this skill, which they might well do if that’s their personality type.
Just typing this out has made me think it’s almost like passing on wisdom and the religious experience are probably inseparable.
Also, many so called prophets can easily exploit this skill, which they might well do if that’s their personality type.
Just typing this out has made me think it’s almost like passing on wisdom and the religious experience are probably inseparable.
I think its because, very often, wisdom requires rewriting a fundamental assumption about the world. And people tend to tie up their identity with their fundamental assumptions.
As a result these changes can destabilize a persons identity, which causes said person to look for the closest source of stability, often the 'peophet'.
This gives the 'prophet' enourmous power over the person, not just because of the identity destabilization, but because when you change someones fundamental assumptions about the world, those changes don't nessecerily have to be truthful or helpful for the person. The person only has to think the changes do.
This identity change and stability dependence is probably what causes the cult like appearance/ behaviour.
As a result these changes can destabilize a persons identity, which causes said person to look for the closest source of stability, often the 'peophet'.
This gives the 'prophet' enourmous power over the person, not just because of the identity destabilization, but because when you change someones fundamental assumptions about the world, those changes don't nessecerily have to be truthful or helpful for the person. The person only has to think the changes do.
This identity change and stability dependence is probably what causes the cult like appearance/ behaviour.
Wow, very well said. Thank you for the insight!
Yet, in the Tantric and Classical view, art was a way that ordinary people can connect to wisdom. No mystical experiencies or psychedelic substances necessary.
What's amazing to me about Christopher Alexander and his work is that he's able to describe the process of generating such art (in the form of architecture) to ordinary people, using plain, relatable language. The links I posted in the earlier comment are examples.
In other words, you don't need to be someone with a rare skill.
What's amazing to me about Christopher Alexander and his work is that he's able to describe the process of generating such art (in the form of architecture) to ordinary people, using plain, relatable language. The links I posted in the earlier comment are examples.
In other words, you don't need to be someone with a rare skill.
In my view, wisdom is just more situational than intelligence. The latter is about abstracting away a problem to its core so that the situation becomes tractable to reason about. With enough abstraction, it becomes easy to write down ten different solutions to a problem, which is then what you find online.
The next step in the process is to undo your abstractions to determine which of the previous solutions (if any) is the best one for your particular situation. It's not entirely identical, but I tend to think of intelligence and wisdom as deductive and inductive parts of the same process.
So I don't think that means that wisdom is so far removed from scientific enquiry that it becomes mystical, far from it: instead, it requires so much more rigour and discipline to codify wisdom into laws that it just doesn't happen that much; and even when it does happen, the most you'll find is fuzzy frameworks on how to deal with certain problems rather than the hard and accurate rules you can find in deductive analysis.
The next step in the process is to undo your abstractions to determine which of the previous solutions (if any) is the best one for your particular situation. It's not entirely identical, but I tend to think of intelligence and wisdom as deductive and inductive parts of the same process.
So I don't think that means that wisdom is so far removed from scientific enquiry that it becomes mystical, far from it: instead, it requires so much more rigour and discipline to codify wisdom into laws that it just doesn't happen that much; and even when it does happen, the most you'll find is fuzzy frameworks on how to deal with certain problems rather than the hard and accurate rules you can find in deductive analysis.
If that is how you conceive of "wisdom", sure, and I can see where you are coming from. It looks at limited notions of wholes (the whole of a chain of logical steps).
However, wholes are nested. The computer you are using to read these words, wherever you are, are part of a larger whole. Further, there is a paradox in which, while parts make up the whole, it's the whole that makes the parts.
Taking those all the way, "The" Whole in which all wholes are parts of, then, is boundless (no edge), and it is beginningless, (no causal origin).
My understanding of the scientific method is that it is ultimately limited in what it can find. It is not necessarily true that the scientific method is capable of explaining everything, though it is broadly applicable. That method is very good for analysis, but not synthesis, and focused on the origin in causal chains rather than the teleos.
However, wholes are nested. The computer you are using to read these words, wherever you are, are part of a larger whole. Further, there is a paradox in which, while parts make up the whole, it's the whole that makes the parts.
Taking those all the way, "The" Whole in which all wholes are parts of, then, is boundless (no edge), and it is beginningless, (no causal origin).
My understanding of the scientific method is that it is ultimately limited in what it can find. It is not necessarily true that the scientific method is capable of explaining everything, though it is broadly applicable. That method is very good for analysis, but not synthesis, and focused on the origin in causal chains rather than the teleos.
[deleted]
As a response to the GP, You're making this much too hard. Google's results have become the crap they describes fairly recently - in the last two years, with a specific and noticeable change (and good stuff is even still there if one works hard and the crappiness might have receded a bit lately, even).
Sure, one also needs understanding to get something out of search. But Google when was in it's sweet-spot, it could get a researcher extra knowledge and insight. After all, a researcher needs both a holistic perspective and information they'd know at the start of an exploration.
Sure, one also needs understanding to get something out of search. But Google when was in it's sweet-spot, it could get a researcher extra knowledge and insight. After all, a researcher needs both a holistic perspective and information they'd know at the start of an exploration.
Some years ago, Google+ had this question for me as a way to populate my profile: "What are things you still cannot find on Google?". I know why they asked that question, yet I was tickled by the more interesting question: "What you will never find on Google?"
I was so tickled that I put it on my profile here on HN and elsewhere -- but all of those things I put down are different ways of saying "Wisdom".
Google has aspirations about organizing the world's knowledge, and making it easy to search for facts and knowledge. What I'm saying here is that people have never been able to find wisdom in a Google search -- and never will. Each person finds it within themselves through awareness of wholes.
So in my view, it is not that it's getting harder to find wisdom from a Google search, but rather, the proliferation of knowledge and facts over the years have increasingly made it even more distracting to find wisdom within themselves.
I was so tickled that I put it on my profile here on HN and elsewhere -- but all of those things I put down are different ways of saying "Wisdom".
Google has aspirations about organizing the world's knowledge, and making it easy to search for facts and knowledge. What I'm saying here is that people have never been able to find wisdom in a Google search -- and never will. Each person finds it within themselves through awareness of wholes.
So in my view, it is not that it's getting harder to find wisdom from a Google search, but rather, the proliferation of knowledge and facts over the years have increasingly made it even more distracting to find wisdom within themselves.
What you'll never find on Google: a good answer to "What should be added to Google?"
The Göögle Incompleteness Theorem
The Göögle Incompleteness Theorem
Yep. Maybe the search for wisdom starts with sitting silent and still under a tree for a while.
Or it's a trick and the mere fact that you're seeking answers and asking for help is the wisest move of all.
Being a practicing Christian who has had several verifiable paranormal, if not supernatural, direct experiences I'm pretty comfortable with mysticism, but I read through that and it just looks like bafflegab. I realize that might be misunderstanding on my part, but I'm the kind of person who thinks C.S. Peirce's semiotic is sensible, and Leibniz's monadology is fascinating, so it's not like I'm unfamiliar with dense philosophy.
> ...several verifiable paranormal, if not supernatural, direct experiences...
As someone who once thought this was true of my experience, yet have since learned they were just common phenomenon, I'm curious what these experiences were. And how do you think they have effected your perspective?
As someone who once thought this was true of my experience, yet have since learned they were just common phenomenon, I'm curious what these experiences were. And how do you think they have effected your perspective?
Not OP, but I can identify with mystical experiences like what's hinted at in that post.
I don't have any experiences I would call "verifiably paranormal", but I've had a few I've not been able to fully account for to my own satisfaction from a purely materialist perspective (despite a total collapse of my childhood evangelical Christian beliefs).
One was sudden, total cessation of a destructive habit I'd struggled to stop for decades, after a single prayer I hadn't realized was related to that struggle ("I don't care what it costs - make me more like You"), a very unexpected turn of improbable events immediately following the prayer, and an intense spiritual experience following the unexpected events.
I have devised pure materialist accountings for it, but they aren't rigorous and haven't persuaded me.
...all that said, I actually found the GP post very understandable and thought it made tremendous amounts of sense - perhaps even was "wise." I certainly wouldn't call it "bafflegab".
I don't have any experiences I would call "verifiably paranormal", but I've had a few I've not been able to fully account for to my own satisfaction from a purely materialist perspective (despite a total collapse of my childhood evangelical Christian beliefs).
One was sudden, total cessation of a destructive habit I'd struggled to stop for decades, after a single prayer I hadn't realized was related to that struggle ("I don't care what it costs - make me more like You"), a very unexpected turn of improbable events immediately following the prayer, and an intense spiritual experience following the unexpected events.
I have devised pure materialist accountings for it, but they aren't rigorous and haven't persuaded me.
...all that said, I actually found the GP post very understandable and thought it made tremendous amounts of sense - perhaps even was "wise." I certainly wouldn't call it "bafflegab".
> despite a total collapse of my childhood evangelical Christian beliefs
I should perhaps clarify that while I had the total collapse and found myself staring nihilism in the face, I currently am drifting towards a very heretical, Bible-is-flawed-and-maybe-not-divine-at-all form of faith in Jesus (in part due to the above-mentioned experience).
I should perhaps clarify that while I had the total collapse and found myself staring nihilism in the face, I currently am drifting towards a very heretical, Bible-is-flawed-and-maybe-not-divine-at-all form of faith in Jesus (in part due to the above-mentioned experience).
The most striking one was an injury transference. I had a severe medically diagnosed intercostal rib strain from a botched deadlift. Someone took it from me by touch. Literal laying on of hands. I mean going from so much pain that rolling over in bed is agony to completely fine. Punchline is she got to have a severe rib injury for a few weeks. I’ve still got no idea how that worked.
I’ve also experienced telepathy, seen a ghost (with multiple corroborating witnesses) and witnessed poltergeist activity. The last of those I can kind of doubt though. The mirror falling off the wall could maybe have been some kind of microquake.
I’ve also experienced telepathy, seen a ghost (with multiple corroborating witnesses) and witnessed poltergeist activity. The last of those I can kind of doubt though. The mirror falling off the wall could maybe have been some kind of microquake.
If you're open to it, I'd love to hear more details and context around these stories.
Email is my profile, if you have the time and inclination to get in touch.
Email is my profile, if you have the time and inclination to get in touch.
Yeah, what you're describing with the injury is a skill I'd call empathic absorption. A rib injury is fairly dramatic and eye-opening. I know someone who did something similar with scarlet fever.
It's good to see this surface up here, though it's the kind of stuff I discuss in other communities.
It's good to see this surface up here, though it's the kind of stuff I discuss in other communities.
I wrote a bunch of stuff, but then I deleted it and decided to just copy someone better at wording than me:
There are more things in Heaven and Earth, Horatio, than are dreamt of in your philosophy
Is this phenomenon reproduceable? Can it be experimentally verified?
[deleted]
I'd have my own share of experiences, so I'll take what you say here at face value. (Though there are a wide variety of experiences).
Given your background, I think you might find Christopher Wallis's Tantra Illuminated -- in particular, the chapter on the View of Tantra -- more sensible. Although it does not talk about wholes, centers, and unfoldings in those terms that Christopher Alexander does.
I think the key is in the non-duality of creator and creation. That within all parts are the Whole. I can't tell from the wikipedia page on Leibniz's monadology if monads are distinct from each other, whereas that stuff about wholes and centers point to apparent distinctions on a continuum that is the Whole.
Given your background, I think you might find Christopher Wallis's Tantra Illuminated -- in particular, the chapter on the View of Tantra -- more sensible. Although it does not talk about wholes, centers, and unfoldings in those terms that Christopher Alexander does.
I think the key is in the non-duality of creator and creation. That within all parts are the Whole. I can't tell from the wikipedia page on Leibniz's monadology if monads are distinct from each other, whereas that stuff about wholes and centers point to apparent distinctions on a continuum that is the Whole.
From [0]:
wisdom, 1. in general, the best form of knowledge.
2. specifically, the intellectual virtue
or science concerning the first or
supreme causes of all things.
philosophical wisdom, intuitive knowledge combined
with scientific knowledge of
the objects which are
naturally the highest;
metaphysics, including
natural theology.
practical wisdom, 1. prudence.
2. any excellent form of
practical knowledge, as of
one of the architectonic
arts.
When I was young, we didn't have StackOverflow or Google. We had to hike through snow to access the internet at all, uphill both ways!
In those days, we would usually read the manual when we needed to dig past a surface-level understanding of how things work. The manuals are dry and dense, but they're much easier to read with modern niceties like "Ctrl+F".
IMO, this is one of the reasons that people recommend using software with a long track record. If you have a question about some parameter in a systemd service script, and the internet doesn't have a ready made answer, the details are all written down in the manual.
https://www.man7.org/linux/man-pages/man5/systemd.service.5....
Plenty of modern tools have comprehensive detailed docs like this. Python, Go, even most widely-used JS frameworks. The primary sources are often downranked in search engines because they don't sell ads, but it's a good idea to find and bookmark them when you start working with a new technology. RTFM!
In those days, we would usually read the manual when we needed to dig past a surface-level understanding of how things work. The manuals are dry and dense, but they're much easier to read with modern niceties like "Ctrl+F".
IMO, this is one of the reasons that people recommend using software with a long track record. If you have a question about some parameter in a systemd service script, and the internet doesn't have a ready made answer, the details are all written down in the manual.
https://www.man7.org/linux/man-pages/man5/systemd.service.5....
Plenty of modern tools have comprehensive detailed docs like this. Python, Go, even most widely-used JS frameworks. The primary sources are often downranked in search engines because they don't sell ads, but it's a good idea to find and bookmark them when you start working with a new technology. RTFM!
>In those days, we would usually read the manual when we needed to dig past a surface-level understanding of how things work. The manuals are dry and dense, but they're much easier to read with modern niceties like "Ctrl+F".
Then, after that didn't work, we had to go to find the weird bearded guy that didn't like to talk to people, but knew everything. He would usually tell you the answer, but in a condescending way that made you understand that you only came to him in "emergencies" and that you were kind of stupid for asking an "obvious" question. Actually, now that I think of it, that is stackoverflow now!
Thank god message boards started taking off! ... but sadly, then the "real" Internet came and killed it. :-(
Then, after that didn't work, we had to go to find the weird bearded guy that didn't like to talk to people, but knew everything. He would usually tell you the answer, but in a condescending way that made you understand that you only came to him in "emergencies" and that you were kind of stupid for asking an "obvious" question. Actually, now that I think of it, that is stackoverflow now!
Thank god message boards started taking off! ... but sadly, then the "real" Internet came and killed it. :-(
Before the real internet came and killed the forums, the weird bearded guy would lurk on the forums. He's the guy who makes you answer general questions like,
"Why would you want to do that?"
"What's this for?"
"Did you try searching before asking such a stupid question?"
"Why would you want to do that?"
"What's this for?"
"Did you try searching before asking such a stupid question?"
Those are the same questions that people ask on SO, although the respondents aren't exclusively men with beards.
OP bemoaned how people on StackOverflow would tell you not to do what you're asking about. Reference material is not the cause of that "weird bearded guy" problem, but it is one possible solution.
We're talking about software developers, not wizards. It's not like you have to offer a dram of blood and draw a pentagram with salt before you approach the ancient grimoires.
OP bemoaned how people on StackOverflow would tell you not to do what you're asking about. Reference material is not the cause of that "weird bearded guy" problem, but it is one possible solution.
We're talking about software developers, not wizards. It's not like you have to offer a dram of blood and draw a pentagram with salt before you approach the ancient grimoires.
Even if they aren’t literally weird old men with beards who want to make you feel bad for bothering them, you know deep inside there lurks one. (Jk - also, have been the cranky one, even if I didn’t have a beard at the time).
>We're talking about software developers, not wizards. It's not like you have to offer a dram of blood and draw a pentagram with salt before you approach the ancient grimoires.
Back then, it seemed a lot like that!
Back then, it seemed a lot like that!
> Those are the same questions that people ask on SO
On the good threads that become helpful, yeah.
On most of it, no, those are not the questions people ask there. They will focus on unrelated marginal issues, try to refuse to answer the question, just assume the reason the person is doing it (even when explicitly told on the question), and just throw wrong answers on the wall to see if they stick. (Granted, throwing things on the wall is a useful way to answer some questions, but not all.)
It's good that Google rewards the first kind of thread, but the second one is what sends people away from the platform.
On the good threads that become helpful, yeah.
On most of it, no, those are not the questions people ask there. They will focus on unrelated marginal issues, try to refuse to answer the question, just assume the reason the person is doing it (even when explicitly told on the question), and just throw wrong answers on the wall to see if they stick. (Granted, throwing things on the wall is a useful way to answer some questions, but not all.)
It's good that Google rewards the first kind of thread, but the second one is what sends people away from the platform.
I've seen SO answerers who don't know the answer to a question try to force it into an "X-Y problem" and steer it until the question turned into something they could answer.
Q: "I'm trying to configure Foo to produce Bar, but it's giving me this error!"
A1: "What are you really trying to do?" [Unsaid: I don't know how to produce Bar either]
A2: "You should not be trying to produce Bar. Instead you should produce Baz. I know how to do that--follow the following steps..."
A3: "Producing Bar will not solve what I imagine your goal is. In the general case, you may need to produce many different results, which I would rather answer about..."
Q: "Uhh, thanks everyone, but I'm just trying to configure Foo to produce Bar."
Q: "I'm trying to configure Foo to produce Bar, but it's giving me this error!"
A1: "What are you really trying to do?" [Unsaid: I don't know how to produce Bar either]
A2: "You should not be trying to produce Bar. Instead you should produce Baz. I know how to do that--follow the following steps..."
A3: "Producing Bar will not solve what I imagine your goal is. In the general case, you may need to produce many different results, which I would rather answer about..."
Q: "Uhh, thanks everyone, but I'm just trying to configure Foo to produce Bar."
There is so much imposter syndrome and insecurity behind those type of answers. I wish we all had better filters online when our fragile egos go into defensive mode.
The Q&A format of Stack Overflow isn't really built for complex questions. It works really well for being an easily searched repository of short, technical solutions to well-bounded technical problems.
The Elixir community talked about this a lot on the forums back when I was active on there, because language rankings look at Stack Overflow questions and Stack Overflow questions proliferate in languages with footguns. "Don't use obviously_named_function(), use obscr_wrd_fcn() instead" fits the SO formula quite well. Well designed languages, Elixir included, don't produce many of those kinds of question/answers. That leaves significantly more room for discourse around architecture and other higher-level conceptual stuff, which doesn't really have one correct answer, and thus is ill-suited for SO, being relegated to the lower-discoverability forums (and worse, the un-discoverable Discord).
The Elixir community talked about this a lot on the forums back when I was active on there, because language rankings look at Stack Overflow questions and Stack Overflow questions proliferate in languages with footguns. "Don't use obviously_named_function(), use obscr_wrd_fcn() instead" fits the SO formula quite well. Well designed languages, Elixir included, don't produce many of those kinds of question/answers. That leaves significantly more room for discourse around architecture and other higher-level conceptual stuff, which doesn't really have one correct answer, and thus is ill-suited for SO, being relegated to the lower-discoverability forums (and worse, the un-discoverable Discord).
My favorite is when they do that, then take the incorrect made-up answer to a question that was not asked to close the question due to: Duplicate of..., Off-topic because..., Needs details or clarity, Needs more focus and Opinion-based ...
Many such cases!
Many such cases!
ha! I see this in the dev/qa dynamic all the time. It also happens between product to dev/qa as well.
lol, this was the pre-Internet problem. It usually went like this...
>"Why would you want to do that?" because it seems like the right thing to do? Is there a better way.
>"What's this for?" the boss wants it
>"Did you try searching before asking such a stupid question?" I read the manual and thought about it a lot (remember this was pre-Internet)
so, there was a lot of "rough" hand-holding to learn the ropes for edge cases.
>"Why would you want to do that?" because it seems like the right thing to do? Is there a better way.
>"What's this for?" the boss wants it
>"Did you try searching before asking such a stupid question?" I read the manual and thought about it a lot (remember this was pre-Internet)
so, there was a lot of "rough" hand-holding to learn the ropes for edge cases.
Documentation now is no where near the quality of older stuff. The GW BASIC manual is awesome. The manual for WordStar, PC-DOS, COBOL-80, and so on... these were marvelous.
The thing is... languages were smaller because they hadn't started the accretion of thousands of libraries and frameworks.
In my experience, every language out there is somewhat easy to learn and master. The ecosystem around it is an insane and ever-growing Katamri Damacy of (largely) crap. We all must know it, must use it, and must contribute to it because no one trusts the work of the individual and only the work of an aggregate of individuals... people often don't even trust their own code.
The thing is... languages were smaller because they hadn't started the accretion of thousands of libraries and frameworks.
In my experience, every language out there is somewhat easy to learn and master. The ecosystem around it is an insane and ever-growing Katamri Damacy of (largely) crap. We all must know it, must use it, and must contribute to it because no one trusts the work of the individual and only the work of an aggregate of individuals... people often don't even trust their own code.
> Documentation now is no where near the quality of older stuff.
I wouldn't make a blanket statement either way, but there are certainly counterexamples to this:
- The mpv manual[1] is a work of art.
- The Arch Linux wiki[2] is a treasure trove of information for not just Arch-specific topics, but Linux in general.
- MDN[3] is the defacto standard for any web documentation.
- The PostgreSQL[4] documentation is quite thorough and high quality.
What I think explains your point are two things:
1. There's just a vast amount of software since those early days. "Software is eating the world", and it's realistically impossible for most of it to be well documented.
2. A lot of information is spread out and produced by users of the software; in books, on blogs, tutorials, forums, videos, etc. Sure, this might be seen as a failure of the software authors to produce good documentation, but many of these resources wouldn't exist if the web didn't make them accessible. In some ways this is better than having a single source of reference, as you can benefit from the collective wisdom of the hivemind, rather than only from what the author thought relevant to document.
[1]: https://mpv.io/manual/
[2]: https://wiki.archlinux.org/
[3]: https://developer.mozilla.org/en-US/
[4]: https://www.postgresql.org/docs/
I wouldn't make a blanket statement either way, but there are certainly counterexamples to this:
- The mpv manual[1] is a work of art.
- The Arch Linux wiki[2] is a treasure trove of information for not just Arch-specific topics, but Linux in general.
- MDN[3] is the defacto standard for any web documentation.
- The PostgreSQL[4] documentation is quite thorough and high quality.
What I think explains your point are two things:
1. There's just a vast amount of software since those early days. "Software is eating the world", and it's realistically impossible for most of it to be well documented.
2. A lot of information is spread out and produced by users of the software; in books, on blogs, tutorials, forums, videos, etc. Sure, this might be seen as a failure of the software authors to produce good documentation, but many of these resources wouldn't exist if the web didn't make them accessible. In some ways this is better than having a single source of reference, as you can benefit from the collective wisdom of the hivemind, rather than only from what the author thought relevant to document.
[1]: https://mpv.io/manual/
[2]: https://wiki.archlinux.org/
[3]: https://developer.mozilla.org/en-US/
[4]: https://www.postgresql.org/docs/
Some software has good documentation nowadays. Back in the day, it was de riguer; no-one would dare to release something without a full manual, if only because they knew it was they who'd end up fielding user questions if they didn't.
a chatgpt program trained on just unix man pages would be interesting.
This is correct. It's because, basically definitionally, the amount of people who have deeply engaged with any topic is very low compared to the number of people who have shallowly engaged with it. This means that the amount of reference material written by people with deep understanding will be much much less than the amount of reference material written by people with shallow understanding.
"Back in the day", this problem was addressed by having significant hurdles in order to publish material on a topic. E.g., books used to be the primary way to learn about a topic, and in order to publish a book you must 1) deeply engage with the material for quite a while simply to write the book and 2) become fairly credentialed / convince a publisher you're worth their time, etc. Same thing with academic papers.
The internet has completely removed all barriers to publishing, meaning it's up to us to sort the wheat from the chaff.
"Back in the day", this problem was addressed by having significant hurdles in order to publish material on a topic. E.g., books used to be the primary way to learn about a topic, and in order to publish a book you must 1) deeply engage with the material for quite a while simply to write the book and 2) become fairly credentialed / convince a publisher you're worth their time, etc. Same thing with academic papers.
The internet has completely removed all barriers to publishing, meaning it's up to us to sort the wheat from the chaff.
> the amount of people who have deeply engaged with any topic is very low compared to the number of people who have shallowly engaged with it
I've been feeling this more and more in relation to software. It feels like the more popular a piece of software is and the more people use it, the harder it is to find and talk to users of said software who have engaged with it past an extremely "shallow" level. I constantly encounter significant, easy to reproduce bugs and issues in very popular software used by millions of people, maybe even tens or hundreds of millions, but if I try to look for any sort of discussion about it online, I usually get nothing.
It feels much easier to discuss software with other real people who use it and think about how they use it when it's something really niche used by maybe a few thousand people at most.
I've been feeling this more and more in relation to software. It feels like the more popular a piece of software is and the more people use it, the harder it is to find and talk to users of said software who have engaged with it past an extremely "shallow" level. I constantly encounter significant, easy to reproduce bugs and issues in very popular software used by millions of people, maybe even tens or hundreds of millions, but if I try to look for any sort of discussion about it online, I usually get nothing.
It feels much easier to discuss software with other real people who use it and think about how they use it when it's something really niche used by maybe a few thousand people at most.
Completely agree with this. Most recently I had this feeling with Unity; I deeply tried to learn and understand it for a few months, only to be stymied at every time by obvious, critical bugs that it didn't seem like anyone else was even encountering.
My suspicion is that most people who engage with Unity are amateur engineers. Nothing wrong with that, of course.
My suspicion is that most people who engage with Unity are amateur engineers. Nothing wrong with that, of course.
Is that really true about books? Wouldn't you just need to convince someone your going to sell enough and how accurate the information is wouldn't be important? Im sure you can find books or textbooks claiming evolution isn't real.
> meaning it's up to us to sort the wheat from the chaff.
I was thinking about this the other day. How do you sort out conflicting information when your not an expert your self. I don't know who to trust sometimes.
> meaning it's up to us to sort the wheat from the chaff.
I was thinking about this the other day. How do you sort out conflicting information when your not an expert your self. I don't know who to trust sometimes.
That used to be at least somewhat true about scientific and technical books. Major publishers protected their brands by hiring editors with some subject matter expertise. Of course, we understood that those books could be outdated, incomplete, or biased but for established publishers the quality was generally fairly good.
In general you talk to other known experts in the field and see what they say.
If you don't find an expert in the field and you publish a crappy book you tend to get a number of reviews (hopefully before you send it out to retail) that your book sucks.
When selling books about fake evolution you're not looking for an expert in the field, you're looking to see if you have an audience that would eat it up.
If you don't find an expert in the field and you publish a crappy book you tend to get a number of reviews (hopefully before you send it out to retail) that your book sucks.
When selling books about fake evolution you're not looking for an expert in the field, you're looking to see if you have an audience that would eat it up.
You as a buyer have publisher's reputation as a filter. There's still chaff but it's a bit easier to avoid.
This is what credentials used to signal, but I’m not sure they’re as reliable anymore…
Yeah, like here on HN the blogs are usually written by people who just learned about the topic and then you go to the comments to see what the people who know more says. People who know seems to not think it is worth it to write a blog about it, but they can comment on posts others write and correct them when they are wrong.
misaligned incentives. people get rewarded for simply publishing SOMETHING. look at companies, bootcamps, whatever the hell. most promotions are given bc you've written some low effort blog post that really add nothing to the broader intelligence of software building. hell, even bootcamps are like "go write/publish something!!" and most of these people have barely a clue as to how software operates. GL sifting through all of that useless content
i wouldn't be surprised if a very large percentage of the last decade of tech blog posts are just remnants of someone's promotion, and is now hyperlinked on some resume, or linkedin page
i wouldn't be surprised if a very large percentage of the last decade of tech blog posts are just remnants of someone's promotion, and is now hyperlinked on some resume, or linkedin page
But how do you know the people in the comments actually know more and aren't just an equivalently low but slightly different common denominator? You don't. The same "shallow knowledge vs deep knowledge" problem still applies.
I often read topics where I am an expert, the comments has better information than the blog in most cases. Comments are written by multiple people since there are corrections from others below the comments etc, that makes them far better than most blogs. Single humans are wrong about a lot, multiple humans arguing are way more accurate.
The case where the hordes of commentors with casual knowledge downvote the one guy who knows enough to have some nuance because his opinion doesn't fit with their simple but wrong understanding seems far more common.
> Single humans are wrong about a lot, multiple humans arguing are way more accurate.
I think this is only true if the commentors are SME's & operating in good faith. We've all seen whole threads hijacked by erroneous groupthink, while the 1 or 2 SME's were buried.
I think this is only true if the commentors are SME's & operating in good faith. We've all seen whole threads hijacked by erroneous groupthink, while the 1 or 2 SME's were buried.
This is true and very unfortunate. My response to this is that when I want to learn anything in-depth, I use textbooks written by experts (e.g. professors, industry experts often from the past). It's hard to verify the expertise of blog authors and SO contributors unlike the authors, and blog posts can often be more about self-promotion than sharing deep insights. Sometimes I even write to these authors asking questions and they actually respond. To some extent, old internet is still alive and beautiful. It's just buried under a lot of noise.
Agree, textbooks are very underrated. The three blue one brown guy had one of the best tips for textbook selection that's been extremely helpful for me. He suggests that textbooks by single authors are the easiest to understand and in retrospect that matches with my experience. Multiple authors tend to refine the work to the point where it's 100% correct but explained in a way that only someone who is already an expert would understand.
Google has been SEOed to death essentially. Little to no valuable content can be found there anymore, beyond exactly what you laid out. The only place I've found to find good, solid content to fill the gap between blog-spam and the sort of deep dive you laid out is technical books.
Granted, there's a lot of chaff in books as well, but their quality to junk ratio is wildly better than Google, and they generally go much deeper.
Granted, there's a lot of chaff in books as well, but their quality to junk ratio is wildly better than Google, and they generally go much deeper.
> Google has been SEOed to death essentially.
It's the nature of the incentive system that they set up. Their only grand vision for the internet was one where Google hit its growth targets every quarter. If you do the math on that, then Google has to be enormous in 2022. Guess what, Google is enormous in 2022, and the only way they could figure out how to do that was to keep growing their ad business. So here we are.
It's the nature of the incentive system that they set up. Their only grand vision for the internet was one where Google hit its growth targets every quarter. If you do the math on that, then Google has to be enormous in 2022. Guess what, Google is enormous in 2022, and the only way they could figure out how to do that was to keep growing their ad business. So here we are.
the only fix is to "pay what you can for good content"
Free means we the users become the commodity.
Free means we the users become the commodity.
Or... And bear with me...
Don't pollute the frigging index with SEO garbage. Google didn't tell everyone to SEO. People didn't even start doing SEO.
Then some arsehole went and had a brainwave about "what if we sold people on tweaking their pages to try to get all our customers fighting each other to show up on the front for the front page"?
I still remember my first time overhearing some SEO guys pitching to somebody while eating lunch. Spat out my food when it dawned on me what I was hearing unfold and started to realize that guy was likely not the only doing that.
The librarian's assistant in me had to be held back that day.
Why? Why would you pollute the Index that way? Monsters... Convincing people poisoning the town well was a good idea...
Don't pollute the frigging index with SEO garbage. Google didn't tell everyone to SEO. People didn't even start doing SEO.
Then some arsehole went and had a brainwave about "what if we sold people on tweaking their pages to try to get all our customers fighting each other to show up on the front for the front page"?
I still remember my first time overhearing some SEO guys pitching to somebody while eating lunch. Spat out my food when it dawned on me what I was hearing unfold and started to realize that guy was likely not the only doing that.
The librarian's assistant in me had to be held back that day.
Why? Why would you pollute the Index that way? Monsters... Convincing people poisoning the town well was a good idea...
> Google didn't tell everyone to SEO.
Google gave lots and lots of ad money to websites that did SEO, so they effectively did tell everyone to SEO - crippling their own search product in favor of their ad machine.
Google gave lots and lots of ad money to websites that did SEO, so they effectively did tell everyone to SEO - crippling their own search product in favor of their ad machine.
Or the entire opposite direction, ensure that it's not possible to get paid for shitty content. Kill the web ads market by legal restrictions to tracking and targeting ads, facilitate ad blockers everywhere.
In this discussion, the people making the good content don't do it for the money, but the people creating spam do. Kill the money, and the commercial SEO crap goes away, and only the enthusiast content stays - as it was in the 'good old days' when simply getting clicks on your site could not get you any money.
In this discussion, the people making the good content don't do it for the money, but the people creating spam do. Kill the money, and the commercial SEO crap goes away, and only the enthusiast content stays - as it was in the 'good old days' when simply getting clicks on your site could not get you any money.
ads market will always be a cat/mouse game, a wild goose chase. you'll never be able to kill it fully.
> the people making the good content don't do it for the money
citation needed. there're two driving forces:
- money
- clout
they're interchangeable. if you have one the other becomes much easier to obtain. if the writer don't do it directly for the money, they're most definitely doing it for the clout.
> the people making the good content don't do it for the money
citation needed. there're two driving forces:
- money
- clout
they're interchangeable. if you have one the other becomes much easier to obtain. if the writer don't do it directly for the money, they're most definitely doing it for the clout.
Or taxes. Imagine if all online transactions had a tax that went to supporting internet infrastructure. I know, I know, couldn't possibly.
Our imaginations have run out because those sweet ad dollars are too good to pass up.
Our imaginations have run out because those sweet ad dollars are too good to pass up.
I can't wait until you can give ChatGPT an ISBN and ask it to answer any question by citing that book. The SNR of a book may be high, but when the cost of finding that signal is having to go through 400 pages of dense material, line by line, word by word... well, Google wins.
So Google Books? [1]
Only problem is that pesky copyright that makes you buy them
[1] https://en.m.wikipedia.org/wiki/Google_Books
Only problem is that pesky copyright that makes you buy them
[1] https://en.m.wikipedia.org/wiki/Google_Books
Google Books is just a search engine. It doesn't yet have the natural language parsing and information synthesis power that chat gpt has
wow. i love this idea
I was trying to find some way to explain to people why google results used to be good (back in the days when signal to noise ratio was much better) and now are crap (combination of endless social media, marketing blogs, etc, and of course google's own need to generate money and hence pushing sites/advertisements to the top). I think you've captured it very succinctly. SEO'd to death, stealing this, thanks!
This. The problem outlined is that of finding that old post which has exactly the right answer, and instead being bombarded with new, not-quite-relevant fluff. That’s not a problem of technology, but of priorities.
Google screwed its search during Eric Schmidt's term with the mentality of "Brands will sort out the mess" to 'combat spam'. The result is a few big sites dominating the search in everything instead of the original results that used to be before those updates. The first infamous one being 'Panda' I believe.
It succeeded in eliminating spam from a lot of small sites and instead resulting in an even bigger spam from the same big sites for every category. Great for people who were doing shoe-shopping at, well, Amazon, probably, but bad for everyone else.
Then they increasingly personalized search results via AI. And that changes your search results so fast that you may not get the same seach results for the same exact keyword in a week's time.
Im increasingly of the opinion that corporate, 'business mentality' should not run any major tech corp. It sees everything in numbers and extreme abstractions, juggling them to make a desirable false reality happen in paper and in metrics, screwing up the real world for every user.
It succeeded in eliminating spam from a lot of small sites and instead resulting in an even bigger spam from the same big sites for every category. Great for people who were doing shoe-shopping at, well, Amazon, probably, but bad for everyone else.
Then they increasingly personalized search results via AI. And that changes your search results so fast that you may not get the same seach results for the same exact keyword in a week's time.
Im increasingly of the opinion that corporate, 'business mentality' should not run any major tech corp. It sees everything in numbers and extreme abstractions, juggling them to make a desirable false reality happen in paper and in metrics, screwing up the real world for every user.
Panda has taught me a lot.
Just before it hit, I had a small site that was on the rise in terms of traffic. A hobbyist site, no ads, all original content, and very much not "thin". An enthusiast site, much like the original internet. Incoming links were organic and spontaneous, from serious sites.
Panda crushed it. Massive pagerank downgrade and traffic decimated to about 5%. In mere hours. A false positive, I suppose. I spent 3 months trying to figure out the reason but nobody, including Google or SEO specialists were able to provide any tangible answer.
That day etched a few important lessons into my long term memory:
1) The size of Google's power. It effectively has the power to decide whether you digitally exist or not. The ultimate traffic controller.
2) Even a good faith move by Google may have 1% of false positives. Which seems totally rational until you realize that at Google's scale this could mean millions of people being totally fucked out of the blue.
3) Worse, when that happens, there's no accountability. You can't go anywhere for help, undo being a false positive, you're just randomly fucked and that's it. You cannot depend on anything ran by Google.
Although I'm fine and it wasn't exactly a pivotal moment in my life, it radically changed my view on Google. Early on, I was a fanboy. Google kicking lame Microsoft's butt with miraculous products like search, maps, gmail. I was cheering them on. As of my "lesson", I've grown increasingly cynical.
That was 10 years ago. They've become infinitely worse.
Just before it hit, I had a small site that was on the rise in terms of traffic. A hobbyist site, no ads, all original content, and very much not "thin". An enthusiast site, much like the original internet. Incoming links were organic and spontaneous, from serious sites.
Panda crushed it. Massive pagerank downgrade and traffic decimated to about 5%. In mere hours. A false positive, I suppose. I spent 3 months trying to figure out the reason but nobody, including Google or SEO specialists were able to provide any tangible answer.
That day etched a few important lessons into my long term memory:
1) The size of Google's power. It effectively has the power to decide whether you digitally exist or not. The ultimate traffic controller.
2) Even a good faith move by Google may have 1% of false positives. Which seems totally rational until you realize that at Google's scale this could mean millions of people being totally fucked out of the blue.
3) Worse, when that happens, there's no accountability. You can't go anywhere for help, undo being a false positive, you're just randomly fucked and that's it. You cannot depend on anything ran by Google.
Although I'm fine and it wasn't exactly a pivotal moment in my life, it radically changed my view on Google. Early on, I was a fanboy. Google kicking lame Microsoft's butt with miraculous products like search, maps, gmail. I was cheering them on. As of my "lesson", I've grown increasingly cynical.
That was 10 years ago. They've become infinitely worse.
> Panda crushed it. Massive pagerank downgrade and traffic decimated to about 5%
Yep. I also had various such legitimate sites. They all sank to oblivion. Worse, I had a small web development outfit - we used to put a link back to our site at the footers of the websites of our clients, like 'Designed by', or 'Developed by' etc. As then-advised by Google as good practice.
All of our clients' websites sunk. Small businesses, small ecommerce sites, blogs, open source project sites. All of them became 'low quality'. Not only their livelihood was sunk by Panda, but also our, the developers' websites were sunk because those sites were linked to us per what Google advised. We lost incoming business. Independent web development, small businesses, blogs etc that were thriving before Google murdered them were forced to use the centralized marketplaces - elance, upwork, amazon etc. Causing the consolidation of the internet and the rise of the algorithm.
Its amazing how no class action lawsuit ensued over that. Google literally backstabbing its own users who adhered to the best practices it advocated, and killing their businesses and livelihoods to amplify shopping searches for Amazon and other corporate brands.
> nobody, including Google or SEO specialists were able to provide any tangible answer.
It was because of 'low quality links' that were pointing to websites. Ie, small sites linking to your site. I remember clients fervently going through all the other websites that they asked to link back to them, and begging them to take off those links so their rankings would improve. Some of them succeeded and rose some in ranking, but nothing near enough to keep their small shops alive, leave aside near anything before Panda.
> 3) Worse, when that happens, there's no accountability.
Yep. One sociopathic old-school corporate exec and a host of totally disconnected corporate executives decided the fate and livelihoods of tens of millions of people directly. Maybe up to hundreds of millions of people if you counted in the business ecosystem that those small businesses supported. Literal feudalism, albeit exercised through private tyrannies without any accountability.
That made me conclude that any corporation that holds the livelihoods of people in its hands has to be tightly regulated by the the democratically elected governments.
> it wasn't exactly a pivotal moment in my life
It was a pivotal moment in the lives of countless people and their families. They lost their businesses, their livelihoods. Even worse, the Internet los the thriving grassroots businesses, communities and creative ecosystem that it had. It may be regaining it with the rise of the creator economy, but that will take time. Nothing can make up for the harm that Google did to the Internet and actual people's livelihoods.
Yep. I also had various such legitimate sites. They all sank to oblivion. Worse, I had a small web development outfit - we used to put a link back to our site at the footers of the websites of our clients, like 'Designed by', or 'Developed by' etc. As then-advised by Google as good practice.
All of our clients' websites sunk. Small businesses, small ecommerce sites, blogs, open source project sites. All of them became 'low quality'. Not only their livelihood was sunk by Panda, but also our, the developers' websites were sunk because those sites were linked to us per what Google advised. We lost incoming business. Independent web development, small businesses, blogs etc that were thriving before Google murdered them were forced to use the centralized marketplaces - elance, upwork, amazon etc. Causing the consolidation of the internet and the rise of the algorithm.
Its amazing how no class action lawsuit ensued over that. Google literally backstabbing its own users who adhered to the best practices it advocated, and killing their businesses and livelihoods to amplify shopping searches for Amazon and other corporate brands.
> nobody, including Google or SEO specialists were able to provide any tangible answer.
It was because of 'low quality links' that were pointing to websites. Ie, small sites linking to your site. I remember clients fervently going through all the other websites that they asked to link back to them, and begging them to take off those links so their rankings would improve. Some of them succeeded and rose some in ranking, but nothing near enough to keep their small shops alive, leave aside near anything before Panda.
> 3) Worse, when that happens, there's no accountability.
Yep. One sociopathic old-school corporate exec and a host of totally disconnected corporate executives decided the fate and livelihoods of tens of millions of people directly. Maybe up to hundreds of millions of people if you counted in the business ecosystem that those small businesses supported. Literal feudalism, albeit exercised through private tyrannies without any accountability.
That made me conclude that any corporation that holds the livelihoods of people in its hands has to be tightly regulated by the the democratically elected governments.
> it wasn't exactly a pivotal moment in my life
It was a pivotal moment in the lives of countless people and their families. They lost their businesses, their livelihoods. Even worse, the Internet los the thriving grassroots businesses, communities and creative ecosystem that it had. It may be regaining it with the rise of the creator economy, but that will take time. Nothing can make up for the harm that Google did to the Internet and actual people's livelihoods.
Thanks for writing that and showing examples with a much more serious impact. It's hard to read.
You make a great point that as of then, the great centralization of the internet began. It's only heads and the tail is cut off. Social media accelerated it even further. This is why 90% of the internet feels dead.
You make a great point that as of then, the great centralization of the internet began. It's only heads and the tail is cut off. Social media accelerated it even further. This is why 90% of the internet feels dead.
Best description I've heard for this is "institutional knowledge". It is not uncommon for a workflow/process to be "well documented", but over the course of time, the people that actually do the process have found small little tweaks that are not part of the original documentation nor do they ever get added to the documentation. Due to this, following the directions will never result in what the current workflow does.
You see this a lot in recipes, and I've heard tales of military being susceptible to this, large manufacturing processes, etc where the tweaks are susceptible to being lost if large cutbacks/layoffs were to affect the people with that tribal knowledge.
You see this a lot in recipes, and I've heard tales of military being susceptible to this, large manufacturing processes, etc where the tweaks are susceptible to being lost if large cutbacks/layoffs were to affect the people with that tribal knowledge.
Its worse than that. Someone posts a good solution on github as a comment -> 50 ppl create a paper about it on medium.com -> 500 copy-paste websites replicate it to their database of shitty ads ridden sites.
Finding something more complex than few cm deep is becoming a hellish task.
I blame Google for lack of moderation and getting their SOE gamed like a little pipsqueek.
Juniors are dying happy coz of how fast they can find trivial answers.
Seniors get shafted. Its so bad that I started to build my own github repo woth various answers I needed over the years.
Finding something more complex than few cm deep is becoming a hellish task.
I blame Google for lack of moderation and getting their SOE gamed like a little pipsqueek.
Juniors are dying happy coz of how fast they can find trivial answers.
Seniors get shafted. Its so bad that I started to build my own github repo woth various answers I needed over the years.
The degree to which most of the Internet's actual intellectual/informational value is still, in 2022, simply providing more-efficient replacement for inter-library loan (that is, book piracy websites), for many fields, has me wondering if it'll ever fully deliver on the potential that many of us used to think it had.
Entry-level content is easier to produce and consume. It is also guaranteed a larger percentage of traffic due to the high influx of newer developers.
Example: it's 2022 and we still somehow upvote blog posts detailing one's stack as if that has any impact on anyone else.
Example: it's 2022 and we still somehow upvote blog posts detailing one's stack as if that has any impact on anyone else.
Stack Overflow is indeed quite painful in this regard, and it really, really doesn't need to be like this. And in fact, it isn't all the time, or in every Stack Exchange. Sometimes, finding the right Stack Exchange site helps a lot (like with retrocomputing), where the premise (say, "I want to get this to run on Windows 95") won't turn into arguments over security patches (or whatever) derailing the question. Or sometimes you just have to find the right people in the community, and nudge them to answer somehow. Certain users tend to be quite generous and go out of their way to answer everything you asked with more to guide you, while others love to chide you and slap you with a question closure or deletion for even thinking about walking off the beaten path.
>Or sometimes you just have to find the right people in the community, and nudge them to answer somehow.
You see this on Reddit all the time. There will be a Q&A sub for a topic dominated by people with scant if any experience who think they're helping people by googling stuff for them and there will be a comedy subreddit for the same topic. Smart people ask questions in the joke sub because they know that having sufficient understanding to be amused by highly contextual stuff weeds out a lot of the casuals.
You see this on Reddit all the time. There will be a Q&A sub for a topic dominated by people with scant if any experience who think they're helping people by googling stuff for them and there will be a comedy subreddit for the same topic. Smart people ask questions in the joke sub because they know that having sufficient understanding to be amused by highly contextual stuff weeds out a lot of the casuals.
I feel you.
I think this is due to the fact that the overall concepts are usually quite easy to understand and therefore easy to talk about.
As soon as you have to dig deeper you need to invest significantly more time to talk about it and your audience becomes really small.
This is amplified by people creating newsletters, content (marketing) in general, but the actual knowledge/wisdom is not given away. It's incredibly frustrating but happens in almost all technological field, especially when they are fast moving.
In mechanical engineering for example a lot of wisdom is also put into norms, standards and so on. Books are written about it. But 10 year old books are still completly valid while in software every couple of years a new framewokr is the new hot shit.
I think this is due to the fact that the overall concepts are usually quite easy to understand and therefore easy to talk about.
As soon as you have to dig deeper you need to invest significantly more time to talk about it and your audience becomes really small.
This is amplified by people creating newsletters, content (marketing) in general, but the actual knowledge/wisdom is not given away. It's incredibly frustrating but happens in almost all technological field, especially when they are fast moving.
In mechanical engineering for example a lot of wisdom is also put into norms, standards and so on. Books are written about it. But 10 year old books are still completly valid while in software every couple of years a new framewokr is the new hot shit.
> But 10 year old books are still completly valid while in software every couple of years a new framewokr is the new hot shit
The more I look at it, the more it feels like software hasn’t had a fundamentally new idea in decades. Frameworks improve, networks get bigger, compute gets cheaper, and we spend most of our time dealing with domain modeling, cache, and statistical inference.
We’ve made great strides in the ease of building and encoded many lessons in our frameworks, scale is bigger too, but the underlying guts of software engineering have been very stable for decades.
The more I look at it, the more it feels like software hasn’t had a fundamentally new idea in decades. Frameworks improve, networks get bigger, compute gets cheaper, and we spend most of our time dealing with domain modeling, cache, and statistical inference.
We’ve made great strides in the ease of building and encoded many lessons in our frameworks, scale is bigger too, but the underlying guts of software engineering have been very stable for decades.
I agree actually...kind of.
And there are good books about software architecture and so on. Unfortunately, for each good one there are at least 50 that are really really bad. There are a lot of technology-related books as well, that teach you about responsive web design with react or stuff like this.
Encoding lessons in frameworks isn't always helpful either in my opinion. Your mental model of the software needs to be correct, otherwise it's just a really painful experience. What works for large scale software is often just pain for beginners.
And there are good books about software architecture and so on. Unfortunately, for each good one there are at least 50 that are really really bad. There are a lot of technology-related books as well, that teach you about responsive web design with react or stuff like this.
Encoding lessons in frameworks isn't always helpful either in my opinion. Your mental model of the software needs to be correct, otherwise it's just a really painful experience. What works for large scale software is often just pain for beginners.
> It's like most of human written content caps out at about the level of description ChatGPT could deliver.
Don't worry, it won't be long until we're lamenting that most of ChatGPT's output caps out at about the level a human could deliver.
Don't worry, it won't be long until we're lamenting that most of ChatGPT's output caps out at about the level a human could deliver.
> Humans know exactly what parts went into it and how it works. Why must wisdom always be squeezed out of a rock?
Partly, because so many developers famously hate writing documentation and hate commenting their code.
If developers followed Donald Knuth's Literate Programming [1] then it would be a big improvement.
People rely on a lot of "tribal knowledge" without ever bothering to write it down, and out of all the fields, software development seems to be particularly notorious for its anti-documentation bias.
[1] https://en.wikipedia.org/wiki/Literate_programming
Partly, because so many developers famously hate writing documentation and hate commenting their code.
If developers followed Donald Knuth's Literate Programming [1] then it would be a big improvement.
People rely on a lot of "tribal knowledge" without ever bothering to write it down, and out of all the fields, software development seems to be particularly notorious for its anti-documentation bias.
[1] https://en.wikipedia.org/wiki/Literate_programming
I feel your pain, and have had similar experiences.
I do feel like wisdom is inherently an "internalize for your own mind" process, and on some level, nobody can do that for you. So of course you will be scrabbling around trying to piece things together until it clicks in your own mind.
I've found having a discussion with another human, who understands the topic, is very helpful. And it is one case where an "academic" setting can be useful, assuming you can actually talk to the professor rather than being forced to parse some unfathomable textbook.
Even that example — a textbook — is similar to what you said. It is literally designed to help you understand a topic. And yet, my experience with them mimics yours. A lot of the material is uselessly general, and a lot of it is uselessly specific and technical. And it almost never answers the specific question you actually want to know at a given moment. So, you have to internalize the information (slowly) and eventually get there.
Maybe there is an "event horizon" of me being able to absorb new material, rather than something external.
I do feel like wisdom is inherently an "internalize for your own mind" process, and on some level, nobody can do that for you. So of course you will be scrabbling around trying to piece things together until it clicks in your own mind.
I've found having a discussion with another human, who understands the topic, is very helpful. And it is one case where an "academic" setting can be useful, assuming you can actually talk to the professor rather than being forced to parse some unfathomable textbook.
Even that example — a textbook — is similar to what you said. It is literally designed to help you understand a topic. And yet, my experience with them mimics yours. A lot of the material is uselessly general, and a lot of it is uselessly specific and technical. And it almost never answers the specific question you actually want to know at a given moment. So, you have to internalize the information (slowly) and eventually get there.
Maybe there is an "event horizon" of me being able to absorb new material, rather than something external.
Well, there's a reason ChatGPT delivers this kind of content. It's what is was trained into, and it's just rewiring the logic in the facts it was given.
(It's very impressive how it rewires the logic, but it can't really know anything, by design.)
And yes, that horizon is very real. It's what the internet rewards, so it's what almost everybody delivers.
(It's very impressive how it rewires the logic, but it can't really know anything, by design.)
And yes, that horizon is very real. It's what the internet rewards, so it's what almost everybody delivers.
This knowledge event horizon is bound to stay around the current position for as long as our primary medium for information is text/video/sound (i.e. linear).
When you google for some programming story you typically look for a story that describes how to combine concept A, B, C and maybe F. The more concepts you add the likelyhood of somebody having described a good implementation story for the combination of those concepts goes down. But you don't need to add many such concepts before the number of potential stories become very large. So for any non trivial number of concepts a blog article is not likely to exist.
We could only really change the game by going from text representation to intelligent representations. Rules or general AI.
However, my main surprise in all this: why do we need this many combinations of concepts in the first place? Why isn't the total domain of things that are interesting to do for utilitarian reasons more limited in size?
When you google for some programming story you typically look for a story that describes how to combine concept A, B, C and maybe F. The more concepts you add the likelyhood of somebody having described a good implementation story for the combination of those concepts goes down. But you don't need to add many such concepts before the number of potential stories become very large. So for any non trivial number of concepts a blog article is not likely to exist.
We could only really change the game by going from text representation to intelligent representations. Rules or general AI.
However, my main surprise in all this: why do we need this many combinations of concepts in the first place? Why isn't the total domain of things that are interesting to do for utilitarian reasons more limited in size?
There are a lot of drivers for fragmentation and opacity online. Venture capital throwing money at start ups to see what sticks, search engine optimization, gut-feeling opinionated blogging to build personal brands (self selecting for unexperienced being loud and misleading a lot of the new arrivals) etc.
The cure for me has been two fold: 1) Focusing on books from reputable publishing houses (curated long form content) - but even that takes filtering. I noticed O'Riley has been putting out a lot of noise lately. 2) Focusing on open-standards, sound engineering. I'd rather focus on understanding hardware, operating systems, networking etc then myriad software as a service offerings full of vendor lock-in dark patterns.
The cure for me has been two fold: 1) Focusing on books from reputable publishing houses (curated long form content) - but even that takes filtering. I noticed O'Riley has been putting out a lot of noise lately. 2) Focusing on open-standards, sound engineering. I'd rather focus on understanding hardware, operating systems, networking etc then myriad software as a service offerings full of vendor lock-in dark patterns.
At this rate we're going to have to go back to getting information on programming computers from books.
On the plus side, that means if you're willing to dig through the research papers and other hard but useful sources, you have an opportunity to make a difference by writing that book.
On the plus side, that means if you're willing to dig through the research papers and other hard but useful sources, you have an opportunity to make a difference by writing that book.
That is kind of exciting though! I'm in the same boat and feeling the frustration of not being able to easily branch swing to the specific information that I want because the path is no longer clear.
That doesn't happen for trivial topics or problems though! It's a sort of new frontier where we have to re-solve potentially solved problems because the solution isn't pervasive enough to be trivial.
We have to push ourselves to overcome a knowledge limit, which throws a nice fat monkey wrench into the idea of having "All human knowledge in the palm of your hand".
There's undiscovered territory out there and it's hard to see unless you get above the trees.
That doesn't happen for trivial topics or problems though! It's a sort of new frontier where we have to re-solve potentially solved problems because the solution isn't pervasive enough to be trivial.
We have to push ourselves to overcome a knowledge limit, which throws a nice fat monkey wrench into the idea of having "All human knowledge in the palm of your hand".
There's undiscovered territory out there and it's hard to see unless you get above the trees.
> After that it always ends in hard to parse research papers, specifications, jumping around the source code of multiple libraries, debugging or reverse engineering. If I get lucky I might land on a 15 year old blog post that no man has seen since inception and it describes exactly what I wanted to know.
What you’re describing takes a lot of time and nerve. Those are two of the foremost requisites of the pursuit of knowledge.
The Web is working. It’s enabling you to even access those old manuals via the Internet Archive, papers via Sci-hub, source code on GitHub and of course that old blog post on page 7 of a DuckDuckGo query.
The Web works y’all!
What you’re describing takes a lot of time and nerve. Those are two of the foremost requisites of the pursuit of knowledge.
The Web is working. It’s enabling you to even access those old manuals via the Internet Archive, papers via Sci-hub, source code on GitHub and of course that old blog post on page 7 of a DuckDuckGo query.
The Web works y’all!
> Why must wisdom always be squeezed out of a rock?
Dictionary says there are 3 components to wisdom: knowledge, experience, and good judgment. Machines can help with the first requirement, knowledge, to an extent but leave you frustrated (as you note). They can even help you dig out wise chestnut buried somewhere (as you did) but then you hit the other two.
This is how it is supposed to work: armed with knowledge, you apply this and gain experience, and after a few close calls the fortunate also develop good judgment, informed by experience of application of knowledge.
You want all that squeezed out how?
Dictionary says there are 3 components to wisdom: knowledge, experience, and good judgment. Machines can help with the first requirement, knowledge, to an extent but leave you frustrated (as you note). They can even help you dig out wise chestnut buried somewhere (as you did) but then you hit the other two.
This is how it is supposed to work: armed with knowledge, you apply this and gain experience, and after a few close calls the fortunate also develop good judgment, informed by experience of application of knowledge.
You want all that squeezed out how?
See also the Preventing the Collapse of Civilization talk by Jonathan Blow
https://youtube.com/watch?v=ZSRHeXYDLko
https://youtube.com/watch?v=ZSRHeXYDLko
Easy, it’s because a lot of(I’m hopeful it’s not all of) so-called values of modern world come from exploitation, negative externalities, and little creative choices.
Working on a honest straight reality is like drinking water, has no calories in it. Coke is better. Wines are more better, and everyone other than than I these days seems to be always on a magic fissile cocktails, and while I don’t get that, they seem well off doing it.
Working on a honest straight reality is like drinking water, has no calories in it. Coke is better. Wines are more better, and everyone other than than I these days seems to be always on a magic fissile cocktails, and while I don’t get that, they seem well off doing it.
[deleted]
The problem is one of the precision of the question and the scope of the answer. Usually some questions are just indicative of a complete lack of understanding of the field, basically making a whole comp science course necessary to answer in depth.
https://www.youtube.com/watch?v=36GT2zI8lVA
https://www.youtube.com/watch?v=36GT2zI8lVA
It seems like SEO and content marketing have long term side effects. I have experienced what you describe, and it’s frustrating.
Have you tried ChatGPT? It gives me better results for a lot of things.
Comparison of ChatGPT and Google on 500 coding queries: https://news.ycombinator.com/item?id=34089423
Comparison of ChatGPT and Google on 500 coding queries: https://news.ycombinator.com/item?id=34089423
I cannot relate to your experience. Python docs are almost always sufficient for my needs. I rarely use Stack Overflow but find it more useful for examples or semantic discussions.
I view documentation as a set facts and stack overflow as additional color.
What type of projects require this workflow?
I view documentation as a set facts and stack overflow as additional color.
What type of projects require this workflow?
I love how quickly after ChatGPT's release we're conflating the chicken and the egg. It's not that the content online is barely better than ChatGPT. ChatGPT is what it is because it was trained on that content.
"Vague and abstract" and "a warning" are the minimum-effort responses. Therefore they will be found in the greatest number.
Most responses are there just to get your attention.
This whole thing runs on attention.
Most responses are there just to get your attention.
This whole thing runs on attention.
The problem here is Google and SEO incentives. Or even perceived SEO gains by writing shallow articles.
Notice how many of the articles have a “Pricing” and a bold “We’re Hiring” link ;-)
Because there's no market for ads targeting one guy fifteen years from now. We've built a web that actively discourages the kind of depth you're after.
When I hit that point I've found that books, textbooks, and YouTube videos can be a good resource.
Because it’s easy to monetize crap. So no one takes the time to reproduce valuable content.
it's because you're confusing knowledge wrappers for knowledge, the payload is always difficult to parse
Try books.
This article is totally ignoring the existence of academic "handbooks", which is where the wisdom lies.
The whole idea is that individual papers are supposed to be exploratory, throwing things at the wall and seeing what sticks. They're supposed to be a deluge of information.
But then every decade or so a team of academics take it upon themselves to serve as editors to a handbook, which attempts to survey the field in terms of history, where the most value has been found so far (and what hasn't panned out), and current promising directions. Usually something like 20-50 chapters, each contributed by a different author.
If you want to get into the wisdom of a field, the first thing you do is pull out the most recent 800-page handbook, read the first few chapters, and then drill down in your area of interest on the remaining part.
To say there "are no prizes for wisdom" is absurd, when being selected to publish in a handbook (or being an editor) is prestigious, a mark that you've very much "made it" in the field.
And of course there are plenty of other things that serve similar roles, such as literature review papers or similar. (In philosophy you can write a Stanford Encyclopedia of Philosophy article, for instance.)
If you aren't finding wisdom anywhere, it means you're simply not looking right.
(And this isn't even to mention the fact that at some point somebody will popularize major progress in a field in a general-audience book, e.g. when Daniel Goleman wrote the book "Emotional Intelligence" or Stephen Hawking wrote "A Brief History of Time".)
The whole idea is that individual papers are supposed to be exploratory, throwing things at the wall and seeing what sticks. They're supposed to be a deluge of information.
But then every decade or so a team of academics take it upon themselves to serve as editors to a handbook, which attempts to survey the field in terms of history, where the most value has been found so far (and what hasn't panned out), and current promising directions. Usually something like 20-50 chapters, each contributed by a different author.
If you want to get into the wisdom of a field, the first thing you do is pull out the most recent 800-page handbook, read the first few chapters, and then drill down in your area of interest on the remaining part.
To say there "are no prizes for wisdom" is absurd, when being selected to publish in a handbook (or being an editor) is prestigious, a mark that you've very much "made it" in the field.
And of course there are plenty of other things that serve similar roles, such as literature review papers or similar. (In philosophy you can write a Stanford Encyclopedia of Philosophy article, for instance.)
If you aren't finding wisdom anywhere, it means you're simply not looking right.
(And this isn't even to mention the fact that at some point somebody will popularize major progress in a field in a general-audience book, e.g. when Daniel Goleman wrote the book "Emotional Intelligence" or Stephen Hawking wrote "A Brief History of Time".)
I agree with you that academic summary works are probably the best way for a non-researcher to learn what exists, what's known (and what isn't). Since i've never seen them discussed or referenced on this website, let me also point out the existence of academic encyclopedias, such as the Springer encyclopedia of algoriths[1] (each entry is essentially a slightly more pedagogical review article about a subfield or important problem in CS, along with loads of references to the literature for digging deeper), and the delightful encyclopedia of distances [2](800 pages long!). A couple others i've seen that may be of interest to this audience are the encyclopedia of systems and control[3], and the encyclopedia of unconventional computing[4]
Unfortunately some of these are absurdly expensive, so if you don't want to go the piracy route the cheapest way to access them is to get a membership to your local public university's library system, which in the US typically costs like $100 a year or something.
[1] https://link.springer.com/referencework/10.1007/978-1-4939-2...
[2] https://link.springer.com/book/10.1007/978-3-662-52844-0
[3] https://link.springer.com/referencework/10.1007/978-3-030-44...
[4] https://link.springer.com/referencework/10.1007/978-1-4939-6...
Unfortunately some of these are absurdly expensive, so if you don't want to go the piracy route the cheapest way to access them is to get a membership to your local public university's library system, which in the US typically costs like $100 a year or something.
[1] https://link.springer.com/referencework/10.1007/978-1-4939-2...
[2] https://link.springer.com/book/10.1007/978-3-662-52844-0
[3] https://link.springer.com/referencework/10.1007/978-3-030-44...
[4] https://link.springer.com/referencework/10.1007/978-1-4939-6...
Not only have I never heard of this kind of "handbook" (in spite of having an advanced degree), it isn't clear to me how they actually would be a reliable source of wisdom. It sounds like they are supposed to be a meta-analysis of the current state field, but to take it up a meta-level, who is analyzing the meta-analysis? How do I know the editors didn't just select their friends who have similar viewpoints? In the abstract, a handbook seems as likely to send me wildly astray as it is to send me down the right path. Almost by design, I'd naively expect handbooks to amplify the status quo and discourage more radical ideas (as most institutions are wont to do). This might be good or bad depending on the status quo but either way I'm likely only going to get out wisdom proportional to what I bring in.
Well, how do you reliably trust anything?
I've never seen a handbook that led anyone "wildly astray". They're put out by major academic publishers (Oxford, Routledge, etc.) who hire (publishing) editors qualified to select qualified (academic) editors to select qualified chapter contributors. It's not like they're randos self-publishing or something.
The entire point is to be a fairly neutral, comprehensive state of whatever field or subfield the handbook covers. And they generally do a pretty good job. A place like Oxford is never going to publish a handbook that's trying to push some ideological agenda and ignoring half the field.
But if you don't trust the senior editors at major academic presses, then I don't know what to tell you.
And since you've never heard of handbooks, see my peer comment with links so you can see they exist. :)
I've never seen a handbook that led anyone "wildly astray". They're put out by major academic publishers (Oxford, Routledge, etc.) who hire (publishing) editors qualified to select qualified (academic) editors to select qualified chapter contributors. It's not like they're randos self-publishing or something.
The entire point is to be a fairly neutral, comprehensive state of whatever field or subfield the handbook covers. And they generally do a pretty good job. A place like Oxford is never going to publish a handbook that's trying to push some ideological agenda and ignoring half the field.
But if you don't trust the senior editors at major academic presses, then I don't know what to tell you.
And since you've never heard of handbooks, see my peer comment with links so you can see they exist. :)
Is that advanced degree an academic oriented one or industry oriented?
People on the academic oriented degrees tend to use them a lot. But industry oriented degrees tend to stick with textbooks. (By the way, yes, textbooks are the other kind where you can find wisdom. Normally in an easier to get, more condensed form, but of an older kind.)
About who selects the books, well, who tells you if a book is any good? Some have very radical untested ideas, others stick to older but proven ones. You decide what book to get.
People on the academic oriented degrees tend to use them a lot. But industry oriented degrees tend to stick with textbooks. (By the way, yes, textbooks are the other kind where you can find wisdom. Normally in an easier to get, more condensed form, but of an older kind.)
About who selects the books, well, who tells you if a book is any good? Some have very radical untested ideas, others stick to older but proven ones. You decide what book to get.
>People on the academic oriented degrees tend to use them a lot.
Are you from the EU? Becuase having been in US STEM graduate programs in 2 different fields I have never come across such a handbook or know anyone who has. It's certainly not common in the US.
Are you from the EU? Becuase having been in US STEM graduate programs in 2 different fields I have never come across such a handbook or know anyone who has. It's certainly not common in the US.
If there are different schools within a field, every one of them might have a handbook (so you might have a handbook on linguistic typology, and on the other hand a handbook on generative grammar - although often the topics are even more narrow), so they still are useful to get a survey of the land even when there are different schools of thought. I also do not at all share your sense that all science is crazy antagonistic and political in the sense that "institutions discourage radical ideas" - maybe that's true of some fields, but definitely not all of them (for example, the idea makes no sense at all for mathematics). Even when different opinions and schools of thought exist that doesn't necessarily mean that there's nothing that people can agree on.
But more concretely, you can just look up the authors that contributed to the handbook and if you do indeed have a degree in the field, you'll probably recognise them and their affiliations and will be able to know (or at least look up) what tradition they belong to and what this implies for the handbook.
But more concretely, you can just look up the authors that contributed to the handbook and if you do indeed have a degree in the field, you'll probably recognise them and their affiliations and will be able to know (or at least look up) what tradition they belong to and what this implies for the handbook.
An example[1] of these academic 800+ pager "handbooks" from Neuroscience:
"Cognitive Neuroscience: The Biology of the Mind" by Gazzaniga et al.
I don't own a physical copy of it yet (only digital), but I do have a copy of an equally outstanding book[2] called "Neuroscience: Exploring the Brain" by Bear et al. An excellent thing about this book is that each chapter has a small section called "Path to Discovery" where leading researchers on a given topic (including Nobel winners such as Eric Kandel) briefly share their story of how they arrived at their discoveries. Another excellent aspect of this book is its rich set of visual illustrations of complex topics. It makes learning a joy. I find their high price justified.
[1] https://wwnorton.com/books/9780393603170
[2] https://www.jblearning.com/catalog/productdetails/9781284211...
"Cognitive Neuroscience: The Biology of the Mind" by Gazzaniga et al.
I don't own a physical copy of it yet (only digital), but I do have a copy of an equally outstanding book[2] called "Neuroscience: Exploring the Brain" by Bear et al. An excellent thing about this book is that each chapter has a small section called "Path to Discovery" where leading researchers on a given topic (including Nobel winners such as Eric Kandel) briefly share their story of how they arrived at their discoveries. Another excellent aspect of this book is its rich set of visual illustrations of complex topics. It makes learning a joy. I find their high price justified.
[1] https://wwnorton.com/books/9780393603170
[2] https://www.jblearning.com/catalog/productdetails/9781284211...
If I am doing academic study the fastest way to find good information is to find couple recent studies and cross match their references for common citations.
This often ends up in summary papers, foundational papers and papers with well founded experiments. The fact as a society we pay for millions of researchers to do (mostly) research is a resource many people ignore.
This often ends up in summary papers, foundational papers and papers with well founded experiments. The fact as a society we pay for millions of researchers to do (mostly) research is a resource many people ignore.
You can pull this trick with academic books, too. If you find such a work in some niche subfield you know little about, but aren't sure that book's a good place to start, odds are good the author will name-drop most of the really important books & authors in the introduction. If you do this with two or three and cross-reference, any mentioned in more than one is probably something you ought to look at.
Could you please give some examples of handbooks in some fields? I have read survey papers but this is the first time I’ve heard about the handbook concept
Routledge and Oxford are major publishers. Browse to your heart's content:
https://www.routledgehandbooks.com/home
https://academic.oup.com/pages/oxford-handbooks
One I've read cover-to-cover, for example, is the OUP Handbook of Political Science:
https://academic.oup.com/edited-volume/35474
https://www.routledgehandbooks.com/home
https://academic.oup.com/pages/oxford-handbooks
One I've read cover-to-cover, for example, is the OUP Handbook of Political Science:
https://academic.oup.com/edited-volume/35474
Maybe the article is highlighting the fact that scientific papers are more accessible to the general public than handbooks: You can easily find academic papers on Arxiv or Google Scholar, newspapers cite them, etc. While handbooks don't get much publicity.
The frustrating part is that excellent content, the type that is relevant, well researched, well written, that stands head and shoulders above anything else is out there. More than ever before.
I know this because I'm subscribed to lots of newsletters in multiple fields. They contain one gem after the other. There's a wealth of top notch content that does contain said wisdom. As in, read just this article and skip the 100 other ones. Because it is so damn good.
But I have to actively store them to notes or bookmarks, because you'll never find a single one in Google.
Google has become completely broken for me. In general it returns low quality verbose content, which seems a variation of keyword spamming from 20 years ago. Gonna be great when those crappy pages can be generated with AI.
Google promotes stale content that is outdated or simply incorrect over anything else. And not only that, this effect strengthens over time.
Google increasingly dismisses your input. You type a few words and it simply drops a few.
Google has no understanding of meaning. You'll type "red flower thailand" and it'll just include flowers from any other random country, including those not even red.
Google indexes everything and still can't figure out an original source. It'll show you spammy Pinterest garbage over the actual high quality source.
So here we are. We do have a treasure chest of wisdom. But wisdom doesn't click ads so fuck us I guess. And if this isn't depressing enough, Google will continue to get away with it. Google only has to work for the masses posing normie surface level questions. It does that. It works.
I know this because I'm subscribed to lots of newsletters in multiple fields. They contain one gem after the other. There's a wealth of top notch content that does contain said wisdom. As in, read just this article and skip the 100 other ones. Because it is so damn good.
But I have to actively store them to notes or bookmarks, because you'll never find a single one in Google.
Google has become completely broken for me. In general it returns low quality verbose content, which seems a variation of keyword spamming from 20 years ago. Gonna be great when those crappy pages can be generated with AI.
Google promotes stale content that is outdated or simply incorrect over anything else. And not only that, this effect strengthens over time.
Google increasingly dismisses your input. You type a few words and it simply drops a few.
Google has no understanding of meaning. You'll type "red flower thailand" and it'll just include flowers from any other random country, including those not even red.
Google indexes everything and still can't figure out an original source. It'll show you spammy Pinterest garbage over the actual high quality source.
So here we are. We do have a treasure chest of wisdom. But wisdom doesn't click ads so fuck us I guess. And if this isn't depressing enough, Google will continue to get away with it. Google only has to work for the masses posing normie surface level questions. It does that. It works.
Pages filled with crappy generated AI are what I get every time I have searched for anything recently.
Which leads people to post what would be their google search to Reddit, after finding any group related to the topic.
Which is leading to Reddit forums being filled with the same beginner questions over and over and over and over...
... from people who don't even bother to join the group.
Which leads people to post what would be their google search to Reddit, after finding any group related to the topic.
Which is leading to Reddit forums being filled with the same beginner questions over and over and over and over...
... from people who don't even bother to join the group.
> I know this because I'm subscribed to lots of newsletters in multiple fields. They contain one gem after the other. There's a wealth of top notch content that does contain said wisdom. As in, read just this article and skip the 100 other ones. Because it is so damn good.
Can you recommend your favorite ones?
I'm particularly interested in startups/tech/programming/AI topics, but if you know some great examples from other fields, please share them as well!
Can you recommend your favorite ones?
I'm particularly interested in startups/tech/programming/AI topics, but if you know some great examples from other fields, please share them as well!
My interests are so specific that if I list them, I'll dox myself. Which isn't a huge deal, but still want to stay anon here. Still, I can throw a bone by including one that is fairly generic: front-end development.
My before situation: I used to follow dozens of experts on Twitter. Which is a continuous effort, and includes lots of noise. And that's just Twitter, I also followed some Medium authors, Youtubers, and sometimes watch videos of conferences (that I did not attend). I replaced all that with a single newsletter:
https://frontendfoc.us/
It comes weekly. If anything of any importance in the field happened, it's in there. It's such a vast efficiency win. Once per week, I spent some 30-60 mins reading it, and I'm up to speed.
Generally, 80-90% of the contents I can quickly scan through as they're not that new or important for me to dig in to. Yet 10% is impactful and bookmarkable, the gems I referred to earlier.
I'd recommend to just try lots of newsletters from your field and let them compete. Read them for a few weeks and keep unsubscribing until you have the ultimate one. Go for weekly newsletters, not daily ones. Let the information come to you instead of chasing it, reserve fixed reading time in your schedule.
Hope this helps!
My before situation: I used to follow dozens of experts on Twitter. Which is a continuous effort, and includes lots of noise. And that's just Twitter, I also followed some Medium authors, Youtubers, and sometimes watch videos of conferences (that I did not attend). I replaced all that with a single newsletter:
https://frontendfoc.us/
It comes weekly. If anything of any importance in the field happened, it's in there. It's such a vast efficiency win. Once per week, I spent some 30-60 mins reading it, and I'm up to speed.
Generally, 80-90% of the contents I can quickly scan through as they're not that new or important for me to dig in to. Yet 10% is impactful and bookmarkable, the gems I referred to earlier.
I'd recommend to just try lots of newsletters from your field and let them compete. Read them for a few weeks and keep unsubscribing until you have the ultimate one. Go for weekly newsletters, not daily ones. Let the information come to you instead of chasing it, reserve fixed reading time in your schedule.
Hope this helps!
Do you have any recommendation for backend development or software engineering in general?
If you're not comfortable with sharing those, feel free not to!
If you're not comfortable with sharing those, feel free not to!
Yeah, this is helpful, thanks for sharing!
I got sick of exactly the complaints you're talking about and moved to Kagi.
I find searching for things more enjoyable now, similar to how I felt in the 90s with Google. However I'm not sure if that's due to a bias in results for sites I prefer. It is significantly less spammy though.
I find searching for things more enjoyable now, similar to how I felt in the 90s with Google. However I'm not sure if that's due to a bias in results for sites I prefer. It is significantly less spammy though.
Could you point us towards some of this excellent content and how you found it?
Not OP but I’ll chime in with a favourite: Matt Levine’s Money Stuff.
It’s fun to read, informative and insightful.
It’s fun to read, informative and insightful.
Please see my answer above.
A key part of a scientific-research education is learning how to filter the vast pool of literature and recover interesting and meaningful books, reviews and primary publications. It's not that trivial and is definitely a learned skill.
Take the author's example, 'entropy'. Well, a one-word search is of course going to generate a massive pile of unsorted material. Internet Archive Scholar gives ~873,000 hits. Nice resource by the way:
https://news.ycombinator.com/item?id=33919823
If you knew nothing about entropy, but knew how to research, you might start with, "okay, when is the first appearance of the word 'entropy' in the literature" and you quickly discover it's Ludwig Boltzmann, 1872. Seems to have something to do with energy and information, and if we add in those terms, we're down to 284K hits. At this point, you might think "I bet someone has written a good book on this broad topic, maybe I shouldn't be looking at the primary literature until later."
Notice that adding search terms narrows results? Keep doing that until you get a smaller number of hits. Now, you can grab the top dozen or so papers, and flip to the bibliography, and look to see if they all reference a landmark paper (this is related to searching by citation count). That's probably one you want to look at.
With non-research-literature-focused internet search engines, the same general rules apply. More search terms tends to get better results. If you find a site with a good article, search that site for more. For example, this gives a lot of good results, for the role of entropy in machine learning concepts at least:
'site:towardsdatascience.com entropy Shannon'
It's basically mining a haystack for needles. There are lots of techniques and strategies one can use to speed up the process.
Take the author's example, 'entropy'. Well, a one-word search is of course going to generate a massive pile of unsorted material. Internet Archive Scholar gives ~873,000 hits. Nice resource by the way:
https://news.ycombinator.com/item?id=33919823
If you knew nothing about entropy, but knew how to research, you might start with, "okay, when is the first appearance of the word 'entropy' in the literature" and you quickly discover it's Ludwig Boltzmann, 1872. Seems to have something to do with energy and information, and if we add in those terms, we're down to 284K hits. At this point, you might think "I bet someone has written a good book on this broad topic, maybe I shouldn't be looking at the primary literature until later."
Notice that adding search terms narrows results? Keep doing that until you get a smaller number of hits. Now, you can grab the top dozen or so papers, and flip to the bibliography, and look to see if they all reference a landmark paper (this is related to searching by citation count). That's probably one you want to look at.
With non-research-literature-focused internet search engines, the same general rules apply. More search terms tends to get better results. If you find a site with a good article, search that site for more. For example, this gives a lot of good results, for the role of entropy in machine learning concepts at least:
'site:towardsdatascience.com entropy Shannon'
It's basically mining a haystack for needles. There are lots of techniques and strategies one can use to speed up the process.
I point people to ESR's "How To Ask Questions The Smart Way" [1], using a search engine (and libraries) is indeed an important skill that must be learned and honed.
I really enjoyed Fravia's [2] SearchLores [3] site on how to dig deep into the knowledge of the internet. Most people rarely cross the boundary of the SEO sludge into the really good stuff.
There are a couple easy patterns, starting from a seed to iteratively widen and deepen the context.
Wikipedia, Archive.org, site:edu, filetype:pdf, libgen, and now chat.openai.com
I just asked chatgpt, 'What is a good home experiment to show the concept of "entropy"' and the answer was excellent.
I then tried the same question on DDG but appended "site:edu" and found https://paradigms.oregonstate.edu/activity/325/
Now I am off to do the "Time Dilation Clock Skit" with my kid
https://paradigms.oregonstate.edu/activity/548/
[1] https://www.catb.org/esr/faqs/smart-questions.html (something wrong with his certs)
[2] https://en.wikipedia.org/wiki/Fravia
[3] https://web.archive.org/web/20191201105758/http://search.lor...
I really enjoyed Fravia's [2] SearchLores [3] site on how to dig deep into the knowledge of the internet. Most people rarely cross the boundary of the SEO sludge into the really good stuff.
There are a couple easy patterns, starting from a seed to iteratively widen and deepen the context.
Wikipedia, Archive.org, site:edu, filetype:pdf, libgen, and now chat.openai.com
I just asked chatgpt, 'What is a good home experiment to show the concept of "entropy"' and the answer was excellent.
I then tried the same question on DDG but appended "site:edu" and found https://paradigms.oregonstate.edu/activity/325/
Now I am off to do the "Time Dilation Clock Skit" with my kid
https://paradigms.oregonstate.edu/activity/548/
[1] https://www.catb.org/esr/faqs/smart-questions.html (something wrong with his certs)
[2] https://en.wikipedia.org/wiki/Fravia
[3] https://web.archive.org/web/20191201105758/http://search.lor...
I'm only commenting because by coincidence, I wrote a similar google search yesterday
-----------------------
The first page of results for
"dan dill entropy site:bu.edu"
Yields
https://www.bu.edu/genchem/ch131-summer-1-2021/notes/SecondL...
-----------------------
The first page of results for
"dan dill entropy site:bu.edu"
Yields
https://www.bu.edu/genchem/ch131-summer-1-2021/notes/SecondL...
> With non-research-literature-focused internet search engines, the same general rules apply. More search terms tends to get better results.
Not anymore. For a decade or so.
Nowadays adding terms on general purpose internet search engines makes your results less focused, not more.
Not anymore. For a decade or so.
Nowadays adding terms on general purpose internet search engines makes your results less focused, not more.
I don't understand your comment. Something like:
'entropy shannon machine learning differential'
will give more focused results than just 'entropy', certainly. I suppose there is some limit, is that what you mean? Even there, if I add 'literature review' the results get more focused.
'entropy shannon machine learning differential'
will give more focused results than just 'entropy', certainly. I suppose there is some limit, is that what you mean? Even there, if I add 'literature review' the results get more focused.
If you add terms until a search engine gets out of results it considers relevant, it will start rewriting your query until it finds relevant results again.
Your examples only work because there are plenty of popular sites that talk about this stuff.
Your examples only work because there are plenty of popular sites that talk about this stuff.
I think what you're addressing here is that search engines are not infinite. I think one of the things that's changed here in the past view years is 'infinite shit SEO engines'. Some of these sites are obvious and you can type random statements and see that some series of garbage sites attempt to give hits on it. There are less obvious versions of these sites, and I'm assuming they are much harder for Google to detect.
Now things boil down to the Chinese room problem. Take a topic with no 'popular' sites that can work as traffic directors. How can Google determine if you're authoritative on the subject, or if you're just a SEO site spitting out spam?
Now things boil down to the Chinese room problem. Take a topic with no 'popular' sites that can work as traffic directors. How can Google determine if you're authoritative on the subject, or if you're just a SEO site spitting out spam?
> The drop in quality and rise in quantity of papers published makes keeping up with the scientific literature both expensive and inefficient in terms of time and energy, which slows down acquisition of knowledge and leaves less time for reflection and gaining experiences that are prerequisites for wisdom. So what incentives are there for a scientist or engineer to aspire to be wise given the lack of prizes and career rewards for wisdom? In Chinese thought wisdom is perceived as expertise in the art of living, the ability to grasp what is happening, and to adjust to the imminent future (Simandan, 2018). All of these attributes seem to be advantageous to a career based on solving problems but you need the sagacity to realise that the rewards are indirect and often intangible.
Knowledge is memorizing information
Smart is being able to absorb and process things quickly
Wisdom is knowing and picking the right course of action
Understanding is knowing why, and what you're trading with this choice
Smart is being able to absorb and process things quickly
Wisdom is knowing and picking the right course of action
Understanding is knowing why, and what you're trading with this choice
Thinking of wisdom like food is a good analogy. But there are no "hunger pangs" when we starve for wisdom and no direct dopamine release when we consume wisdom (without training). The reward of wisdom is a better and happier life, and may take weeks / months to reveal its benefits unlike the immediate pleasure of eating. So few people care about it because it takes vision and patience, and does not reveal itself immediately.
More or less, people generally don't seek wisdom because they have poor vision for the future.
More or less, people generally don't seek wisdom because they have poor vision for the future.
Wisdom has never been popular, the dynamics that led to the death of Socrates have continued from his day to our own.
I find it interesting that many different philosophical traditions all seem to discover an ascetic ideal in one way or another. Maybe "ideal" is not the right word. I think if you really examine your own life and existence one would have to admit that they would be much better off if they could learn to be happy with an sparse lifestyle. But as you point out, this is not a very popular proposition either.
I was going to mention this exact thing. I will add not only do humans not want wisdom, but they also do not like to:
learn from history (or even study it)
learn from elders (or even be around them)
read books
etc
I do realize that HN is subset of the population who do enjoy much of these things, but HN self-selects by nature of content.
learn from history (or even study it)
learn from elders (or even be around them)
read books
etc
I do realize that HN is subset of the population who do enjoy much of these things, but HN self-selects by nature of content.
Ergo the shadowban.
People who are shadowbanned don't want to hear it, but they almost always are banned for being jerks, trolls, and/or propagandists. It's not because they said "the truth" and nobody wanted to hear it; it's because nobody wanted to hear them because of the way they're acting.
But of course they all think that they're martyrs for speaking the truth.
But of course they all think that they're martyrs for speaking the truth.
Some people get shadowbanned because they are right but too obnoxious, some people because they are too obnoxious and not even right. The second group tends to think they are in the first group.
Oh, we poor cognoscendi, forever living in the world of resentful ignoramuses?
Perhaps taking the example of Plato, a starting-point for wisdom is to try to define your terms, in part through recalling what others have said.
Take e.g., the citation "In Chinese thought wisdom is perceived as expertise in the art of living, the ability to grasp what is happening, and to adjust to the imminent future (Simandan, 2018)."
Aristotle and Plato would call this phronesis, practical wisdom, where the root phrein refers to the gut -- where, not co-incidentally, Zeus put/ate his wife and gained by her wisdom (and modern-day scientists eagerly study the GI nervous system's role in anxiety). Aristotle rooted that in understanding politics (vagarities in how people react), economics/incentive systems, and of course the physical world in terms of knowledge-required, but in light of the recent well-educated democratic leaders who became tyrants, he posed it mainly as a question of character, not knowledge. (Remember Aristotle left his home to become the tutor of Alexander the Great.)
Some related terms from that time and place...
Nous: pure thought, thought considering itself, typically as validation for principles and coherence of chains of reasoning. Quite similar to Descartes' notion of the irreducibility of the sense of one's own mental activity, combined with the pureness of its continuity (that must of course be grounded in God).
Dianoia: two-thought, logical and a dialectical thought depending on reasoning chains from point to point. (cf Paranoia, i.e., concurrency dianoia, and Parmenides: "Mortals wander two-headed")
Aisthesis: perception, awareness.
Pistis: belief
Doxa: opinion
Episteme: understanding (standing around, or around the pillar), scientific reasoning, from facts with an account from principles. nb Theaetetus' initial stab at defining Episteme: "as far as I can see at present, episteme is nothing other than aisthesis" - i.e., all knowledge is rooted in perception or all knowledge is a kind of perception, depending on whether you're empiricist.
Most interesting is the term Sophrosyne, which is untranslatable. Sometimes wisdom, sometimes charity, later chastity. It's the basis for the Delphic maxim, "Know thyself". The ability to stand your post, to know what you know and what you don't know. Exemplified by Socrates in the calmness of his retreat at the loss of Potidea, where he saved others by not losing his head (by contrast to the virtue of courage, the ability to move forward notwithstanding danger and fear). If you want to investigate why we don't privilege wisdom, you could start by seeing why Sophrosyne cannot be translated.
Worth mentioning is Parmenides' (much earlier) idea that the philosopher is the person who knows his way through every town. In an era when the Mediterranean world was transformed by openness to trade and other societies, as grounded in the religious obligation to welcome strangers, and when Greeks defined themselves in part through the legend of Odysseus wandering before he returned home after war (another Sophrosyne story), it's somewhat appropriate to our own era.
Theaetetus (who produced a mathematical proof of irrational numbers) gave his definition after feeling completely lost. Socrates replied by saying all philosophy begins in wonder. So, is having a question, or starving for wisdom, is a kind of hunger, "resolved" with knowledge, so all we have to do is define our terms?
After Socrates took the hemlock and lay dying, Plato describes his death rather graphically, as his feet getting firm, then his legs, his body, etc until he was fully fixed. The language used is exactly that used for "defining" terms, suggesting that the process of definition itself is a kind of death. It certainly kills wonder :)
Perhaps taking the example of Plato, a starting-point for wisdom is to try to define your terms, in part through recalling what others have said.
Take e.g., the citation "In Chinese thought wisdom is perceived as expertise in the art of living, the ability to grasp what is happening, and to adjust to the imminent future (Simandan, 2018)."
Aristotle and Plato would call this phronesis, practical wisdom, where the root phrein refers to the gut -- where, not co-incidentally, Zeus put/ate his wife and gained by her wisdom (and modern-day scientists eagerly study the GI nervous system's role in anxiety). Aristotle rooted that in understanding politics (vagarities in how people react), economics/incentive systems, and of course the physical world in terms of knowledge-required, but in light of the recent well-educated democratic leaders who became tyrants, he posed it mainly as a question of character, not knowledge. (Remember Aristotle left his home to become the tutor of Alexander the Great.)
Some related terms from that time and place...
Nous: pure thought, thought considering itself, typically as validation for principles and coherence of chains of reasoning. Quite similar to Descartes' notion of the irreducibility of the sense of one's own mental activity, combined with the pureness of its continuity (that must of course be grounded in God).
Dianoia: two-thought, logical and a dialectical thought depending on reasoning chains from point to point. (cf Paranoia, i.e., concurrency dianoia, and Parmenides: "Mortals wander two-headed")
Aisthesis: perception, awareness.
Pistis: belief
Doxa: opinion
Episteme: understanding (standing around, or around the pillar), scientific reasoning, from facts with an account from principles. nb Theaetetus' initial stab at defining Episteme: "as far as I can see at present, episteme is nothing other than aisthesis" - i.e., all knowledge is rooted in perception or all knowledge is a kind of perception, depending on whether you're empiricist.
Most interesting is the term Sophrosyne, which is untranslatable. Sometimes wisdom, sometimes charity, later chastity. It's the basis for the Delphic maxim, "Know thyself". The ability to stand your post, to know what you know and what you don't know. Exemplified by Socrates in the calmness of his retreat at the loss of Potidea, where he saved others by not losing his head (by contrast to the virtue of courage, the ability to move forward notwithstanding danger and fear). If you want to investigate why we don't privilege wisdom, you could start by seeing why Sophrosyne cannot be translated.
Worth mentioning is Parmenides' (much earlier) idea that the philosopher is the person who knows his way through every town. In an era when the Mediterranean world was transformed by openness to trade and other societies, as grounded in the religious obligation to welcome strangers, and when Greeks defined themselves in part through the legend of Odysseus wandering before he returned home after war (another Sophrosyne story), it's somewhat appropriate to our own era.
Theaetetus (who produced a mathematical proof of irrational numbers) gave his definition after feeling completely lost. Socrates replied by saying all philosophy begins in wonder. So, is having a question, or starving for wisdom, is a kind of hunger, "resolved" with knowledge, so all we have to do is define our terms?
After Socrates took the hemlock and lay dying, Plato describes his death rather graphically, as his feet getting firm, then his legs, his body, etc until he was fully fixed. The language used is exactly that used for "defining" terms, suggesting that the process of definition itself is a kind of death. It certainly kills wonder :)
There's knowledge and then there's understanding.
Understanding is far more useful, but (I think) less valued.
Knowledge can be gamed, you can know things that simply aren't true.
Understanding can be used not only to extract new knowlegde from a corpus of data, but also as a means to validate things presented as fact.
Understanding is far more useful, but (I think) less valued.
Knowledge can be gamed, you can know things that simply aren't true.
Understanding can be used not only to extract new knowlegde from a corpus of data, but also as a means to validate things presented as fact.
"information" is far too generous. We are drowning in data. Data is
many degrees lower in the food chain than "information" (what is
informative), which is a world apart from "wisdom".
When working with our enterprise customers (we make a platform for commercial vehicles), I frequently hear "we want as much data as you can give us"
Which I tend to politely respond to with "You want insights. Unless you have a means to effectively extract those from all that data we can give you, that data is just a liability for you."
Most data is noise. Finding the needle in the haystacks (or the patterns that are actually the needles more likely) is. where all the value is.
Which I tend to politely respond to with "You want insights. Unless you have a means to effectively extract those from all that data we can give you, that data is just a liability for you."
Most data is noise. Finding the needle in the haystacks (or the patterns that are actually the needles more likely) is. where all the value is.
> So what incentives are there for a scientist or engineer to aspire to be wise given the lack of prizes and career rewards for wisdom?
If everything comes down to incentives at the end of the day, it defeats any sense of morality. Perhaps that's the author's point here. The wise scientist/engineer does what they believe is right and the rest is history. In their attempt for doing what's right, they may accidentally create their best work.
This overall reminds me of the search for the philosopher's stone. Where many ended up finding "philosophical gold(wisdom)" through the process of trying to find it and lead to their own individual magnum opus(great work).
https://en.wikipedia.org/wiki/Philosopher%27s_stone
https://en.wikipedia.org/wiki/Masterpiece
If everything comes down to incentives at the end of the day, it defeats any sense of morality. Perhaps that's the author's point here. The wise scientist/engineer does what they believe is right and the rest is history. In their attempt for doing what's right, they may accidentally create their best work.
This overall reminds me of the search for the philosopher's stone. Where many ended up finding "philosophical gold(wisdom)" through the process of trying to find it and lead to their own individual magnum opus(great work).
https://en.wikipedia.org/wiki/Philosopher%27s_stone
https://en.wikipedia.org/wiki/Masterpiece
"[Doing] what they believe is right" is one kind of incentive as not all incentives are monetary.
Ah, to heck with wisdom!!
We are drowning in data while starving for information!!
We are drowning in data while starving for information!!
I think you’re getting downvotes for the phrasing, but this is an insightful point:
We have signal; which only becomes information when you successfully decode it.
Which raises questions: eg, would we be better off sampling less and analyzing more — to extract a better portion of information from the already captured signal?
But the confusion itself also points to a lack of wisdom — lots to think about.
We have signal; which only becomes information when you successfully decode it.
Which raises questions: eg, would we be better off sampling less and analyzing more — to extract a better portion of information from the already captured signal?
But the confusion itself also points to a lack of wisdom — lots to think about.
there's also a lot of noise
If you look at the system as a whole, at least when it comes to search engines, you would even say it demands noise. If there were no potential for monetary gains in SEO spam sites then the amount of noise groups like Google would have to filter would drop dramatically.
This issue has been talked about quite a bit over the past few years and to me it comes down fundamentally to incentives academics have to keep churning papers because that's what they get measured on. Of course the good thing about that is that you can change the incentives, just as how science funding changed significantly post WW2, but I'm afraid just like in many problematic areas of society nowadays there is a large group of people who benefit from this situation and are hence hostile to any sort of change. As such if change does come I think it will have to be top down rather than bottom up.
For an interesting perspective the physicist Murray Gell-Mann spoke on similar issues in a 1997 interview:
https://www.youtube.com/watch?v=tQGjsWiA_mM
For an interesting perspective the physicist Murray Gell-Mann spoke on similar issues in a 1997 interview:
https://www.youtube.com/watch?v=tQGjsWiA_mM
After playing with chatGPT, I'm thinking one solution would be to have chatGPT be trained on all these scientific papers, and then we can just ask it any questions we may have. Obviously, any answers would be thoroughly vetted, but it may be able to drastically reduce the time from paper to wisdom.
Facebook tried this:
https://galactica.org/
Apparently they took it offline after seeing that it generates fake information: https://www.vice.com/en/article/3adyw9/facebook-pulls-its-ne...
Apparently they took it offline after seeing that it generates fake information: https://www.vice.com/en/article/3adyw9/facebook-pulls-its-ne...
Very interesting. Did they use GPT-3?
For me, the job of curators is much more important now than ever.
Sure algorithms make a wonderful job at profiling us and showing us as much content corresponding to our tastes as possible.
But the question of discovering new things is still not solved by algorithms.
Whereas a proper teacher, TV/radio show runner, or newspaper really can make a tremendous job at extending our perspectives.
Second thing, the repetition process. It is very important that we are exposed to powerful ideas, concepts SEVERAL times. And this is something that is not yet properly established in our everyday life. Advertising as a repetition we suffer from should be superseded by proper repetition of more enpowering activities.
Second thing, the repetition process. It is very important that we are exposed to powerful ideas, concepts SEVERAL times. And this is something that is not yet properly established in our everyday life. Advertising as a repetition we suffer from should be superseded by proper repetition of more enpowering activities.
I don't know this article grasps what wisdom is, except for being able to shallowly write out a definition.
There's tons of wisdom out there, but the more widely applicable it is (across different people/experiences), the more abstract or general it must to be and is thus not directly applicable.
Specific wisdom (like for a highly constrained problem) is awesome, but you and your life are quite unique so this is very hard to find or probably doesn't exist. (Should I move away from home for a job? Well.. that depends on a thousand factors AND your unique self)
Trying to optimize for wisdom is a little like optimizing for meaning, I think. Adopting it from others can be counter productive.
There's tons of wisdom out there, but the more widely applicable it is (across different people/experiences), the more abstract or general it must to be and is thus not directly applicable.
Specific wisdom (like for a highly constrained problem) is awesome, but you and your life are quite unique so this is very hard to find or probably doesn't exist. (Should I move away from home for a job? Well.. that depends on a thousand factors AND your unique self)
Trying to optimize for wisdom is a little like optimizing for meaning, I think. Adopting it from others can be counter productive.
IMO there is a missing middle level here : Between Information overload and (real) wisdom, there is synthesised knowledge. I don't believe AI will be "wise", But AI will definitely solve/assist synthesis, which is a big part of the issue discussed here.
This of course does not solve any of the underlying causes (incentives structure..) or the "real" wisdom scarcity itself. How do you solve these in a society where quantity reigns supreme? You can't.
This of course does not solve any of the underlying causes (incentives structure..) or the "real" wisdom scarcity itself. How do you solve these in a society where quantity reigns supreme? You can't.
One is only starving for wisdom if one refuses to eat!
Kong Fu Zi say:
"By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest."
Drowning in information? Youre gaining wisdom the bitter wa ;-)
Starving for wisdom? You may be thinking it is too easy to find wisdom. Try a bit harder. Wisdom might be right behind you when you are focused on how bad something in front of you may seem.
Kong Fu Zi say:
"By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest."
Drowning in information? Youre gaining wisdom the bitter wa ;-)
Starving for wisdom? You may be thinking it is too easy to find wisdom. Try a bit harder. Wisdom might be right behind you when you are focused on how bad something in front of you may seem.
Intelligence is what you know, and wisdom is knowing how to use what you know.
It's intelligent to know how to start a fire and wisdom to know where NOT to start one.
We've got enough intelligence and enough wisdom. What we also have is collective, weaponized purposeful ignorance. People who know better but believe that by banding together with other people, they can force society to abandon both their intelligence and their wisdom.
It's intelligent to know how to start a fire and wisdom to know where NOT to start one.
We've got enough intelligence and enough wisdom. What we also have is collective, weaponized purposeful ignorance. People who know better but believe that by banding together with other people, they can force society to abandon both their intelligence and their wisdom.
In any case we've been always starving for insights
that's true, but makes you wonder how much having more information helps in gaining insight. I think many hold this as a fact that's always true
The quality of the sources, based on your experience with them, might be the point?
A lot of stats or insights I see for things I often wonder how important they really are. Are they all really moving the needle?
What drives me nuts about the usage of statistics in common discourse is that it is used prescriptively instead of descriptively.
E.g. lets say 50% of gun owners accidentally shoot themselves. Now lets say Im talking to someone about how I want to buy a gun for home defense. They tell me "well 50% of gun owners accidentally shoot themselves, so you're safer if you dont have a gun"
This is a gross misunderstanding if statistics. There is no way whatsoever to take the observed general trend, and apply it to a future possibioity in a single instance. What this interpretation misses is that there are variables well within your control. A gun owner who follows all safety rules will virtually never shoot themselves accidentally. Turns out 50% of gun owners are stupid as hell, and you are in control of yiur own outcomes.
I see this problem CONSTANTLY in debates online, on the news, etc. It is very frustrating and contributes greatly to propogating dumb ideas.
E.g. lets say 50% of gun owners accidentally shoot themselves. Now lets say Im talking to someone about how I want to buy a gun for home defense. They tell me "well 50% of gun owners accidentally shoot themselves, so you're safer if you dont have a gun"
This is a gross misunderstanding if statistics. There is no way whatsoever to take the observed general trend, and apply it to a future possibioity in a single instance. What this interpretation misses is that there are variables well within your control. A gun owner who follows all safety rules will virtually never shoot themselves accidentally. Turns out 50% of gun owners are stupid as hell, and you are in control of yiur own outcomes.
I see this problem CONSTANTLY in debates online, on the news, etc. It is very frustrating and contributes greatly to propogating dumb ideas.
There's a risk of considering oneself better than average and that aggregate statistics don't apply to you. Like most drivers consider them better drivers than average. I'd bet the same with gun owners or any other activity. You don't think you can repeat the stupid mistakes, but we all can, you get stressed, sick, drunk,... life is long and full opportunities to make a mistake and get you back to the average case.
By buying a gun and keeping it in your house you move yourself from a population basin of “I will never shoot myself” to one where “If I follow these dozen rules perfectly I will probably never shoot myself.”
You go from a stable basin to an astable mountaintop basin where you are responsible for the maintenance of the banks.
You go from a stable basin to an astable mountaintop basin where you are responsible for the maintenance of the banks.
I'm sure you can appreciate that both imperfect causal inference from statistics and the belief that one is somehow "special" (or exempt from accident and disaster if one simply "puts in the effort") are pervasive forms of human folly that cause a lot of human misery.
Lots of humans are special though and don't have the same problems or weaknesses as average people. For example, some poor people manage to not get fat in USA, if you are one of those then likely many other things that applies to other poor people don't apply to you.
"Prescriptive" statistics in the sense you're describing are probabilities.
And they're both used and useful much of the time. The field of risk management and industry of insurance are based on it.
Your frustrations are not uncommon. They are, however, rather poorly-founded.
They're an excellent illustration failing to get a handle on this safety thing (where "safety" is the inverse of "risk", as one of yesterday's submissions addresses:
<https://news.ycombinator.com/item?id=34036978>
There's also the philosophical question around the "sea battle" question, in which the truth of a statement about a future possibility ("there will be a sea battle tomorrow") is assessed.
<https://en.wikipedia.org/wiki/Problem_of_future_contingents#...>
And they're both used and useful much of the time. The field of risk management and industry of insurance are based on it.
Your frustrations are not uncommon. They are, however, rather poorly-founded.
They're an excellent illustration failing to get a handle on this safety thing (where "safety" is the inverse of "risk", as one of yesterday's submissions addresses:
<https://news.ycombinator.com/item?id=34036978>
There's also the philosophical question around the "sea battle" question, in which the truth of a statement about a future possibility ("there will be a sea battle tomorrow") is assessed.
<https://en.wikipedia.org/wiki/Problem_of_future_contingents#...>
People ask each other where they're from, even though it gives you precisely zero amount of concrete information about a person, and then proceed to paint a picture going forward of that person based on assumptions about other people from there. They also do it to show interest, but it doesn't stop the painting.
Mostly I do it to make conversation. I am almost never asking the question of random strangers, and probably in most cases the social context tells me more about the person than their growing up in Winnetka or Brookline would.
It is true that fifty years ago many people in Denver (and no doubt not only Denver) regarded New York as the embodiment of The Other and were very quick to pick up on a New York accent. But this was silly.
It is true that fifty years ago many people in Denver (and no doubt not only Denver) regarded New York as the embodiment of The Other and were very quick to pick up on a New York accent. But this was silly.
So you say where you grow up has zero influence on who you are?
No, it does. I'm saying asking someone where they're from yeilds no facts about that person if all you get back is a name of the country. It doesn't mean they definitely grew up there, or that they hold citizenship. But it's easy to just take the name of the country and assume a bunch of things about the person based on things you heard about people from there. There may be statistical probabilities there but no certainty. People have done it since forever and it's essentially the same mechanism as OP described.
I am on a community committee for a safe injection site.
They have shown us a graph showing more people are using the site.
I have asked for the same chart to include a metric showing that more people are using the site and there are less overdoses or medical interventions or another metric showing more people using the site results in some sort of quantifiable metric of community benefit.
The public health authority who provides the graphs is unable or unwilling to provide the information and charts I want.
I thus worry that data is selectively used to prove a predetermined narrative.
They have shown us a graph showing more people are using the site.
I have asked for the same chart to include a metric showing that more people are using the site and there are less overdoses or medical interventions or another metric showing more people using the site results in some sort of quantifiable metric of community benefit.
The public health authority who provides the graphs is unable or unwilling to provide the information and charts I want.
I thus worry that data is selectively used to prove a predetermined narrative.
Reaching mass understanding? Innumeracy is everywhere I look. Stats are not moving the needle.
On the other hand, you can use stats for personal gain. For instance, prediction markets. That might have an impact.
On the other hand, you can use stats for personal gain. For instance, prediction markets. That might have an impact.
I imagine that AI used to generate text (e.g ChatGPT) might exacerbate this problem. As the article points out — Some researchers might care more about ‘remixing’ published papers in order to drown the competition, get jobs & other forms of leverage. Generative text makes it much easier
I’ve been thinking of creating a startup founded with the mission to organize the world's information and make it universally accessible and useful. Since there isn’t any other company doing this, I think it could really take off.
... How would you monetize it?
Wikipedia is the closest thing to what you describe, and they have to petition for donations yearly.
Wikipedia is the closest thing to what you describe, and they have to petition for donations yearly.
I have been saying for a long time now that we live in a world awash with answers - I can get you an answer on almost anything.
But the problem is, did you ask the right question (for it to be a useful answer)?
But the problem is, did you ask the right question (for it to be a useful answer)?
Information: Here are some facts
Wisdom: What facts are missing and why do these facts matter?
Wisdom: What facts are missing and why do these facts matter?
Drowning in information... Starving for wisdom...
No matter what we come up with, we still have to seek wisdom from the wise. The internet (AI powered or not) will never be wise.
This gives me hope for humanity, and makes me love God more.
No matter what we come up with, we still have to seek wisdom from the wise. The internet (AI powered or not) will never be wise.
This gives me hope for humanity, and makes me love God more.
> The internet (AI powered or not) will never be wise.
Remember when everyone was all about "the wisdom of crowds," when talking about social media? Yeah, that isn't something anyone says or thinks about now. How quickly did that meme die? And if AI is built from that data, by definition, it cannot be wise.
As Solomon realized, wisdom is a very powerful virtue.
Remember when everyone was all about "the wisdom of crowds," when talking about social media? Yeah, that isn't something anyone says or thinks about now. How quickly did that meme die? And if AI is built from that data, by definition, it cannot be wise.
As Solomon realized, wisdom is a very powerful virtue.
"Comparing information and knowledge is like asking whether the fatness of a pig is more or less green than the designated hitter rule." -- David Guaspari
Don't we all know this? I would love to see solutions than a rewording of the problems that we all are already facing :-).
The problem with this is that it’s hard to blog about it without making yourself a hypocrite.
This situation was posited very early on in the age of the Internet. When everyone can publish, the noise floor rises above everything else. Back in the late 90's early naughts I believed that a new "job/career" would arise which was simply "curator of good stuff related to <x>". There was a lot of discussion about that, and for a long time Google's page rank was a kind of side effect. People would bookmark pages that they found reasonable and then Google would harvest that curation signal and rank that content more highly. In 2010 the book "Building Web Reputation Systems" was published by O'Reilly and a consensus built around "algorithmic curation" based on social cues (like "reputation").
This web site is sort of the epitome of that way of thinking. It's diverse members "vote" on topics and bring them to the front page (or keep them on the front page if they are put there for reasons). And it suffices for a lot of things, but it doesn't really do the job that an editorial staff could do. There is no way to distinguish between votes up or down because the reader agrees or disagrees with the politics of a piece, or the merits. Everyone has their own internal metric for what deserves "up" votes and what deserves "down" votes. The aggregate of those metrics may rarely converge on a consistent notion of "good" or "not good."
At the same time, there is no straight forward way to monetize curation in order to pay the curator(s). Historically that has been via subscriptions to publications, but the resistance to paywalls and intrusive advertising (both mechanisms in use on today's network) have reduced the effectiveness of those techniques tremendously. Worse, the term "blog spam" is a pejorative for a reason, the blog owner, wrapping something they felt was worth reading in some prose and advertisements, do not compensate the author, and for many provide insufficient value to justify their attempts at monetization.
Getting back to the article's main point however, for specialized topics, finding that curation is difficult, getting paid to facilitate it more so. And that means that quality information (wisdom) is drowned in the sea of noise (hot takes).
My guess is that the Overton Window for paying for the work someone has done to distill the wisdom out of the signal has to move from the "radical" to the "acceptable" before we'll see much progress here.
This web site is sort of the epitome of that way of thinking. It's diverse members "vote" on topics and bring them to the front page (or keep them on the front page if they are put there for reasons). And it suffices for a lot of things, but it doesn't really do the job that an editorial staff could do. There is no way to distinguish between votes up or down because the reader agrees or disagrees with the politics of a piece, or the merits. Everyone has their own internal metric for what deserves "up" votes and what deserves "down" votes. The aggregate of those metrics may rarely converge on a consistent notion of "good" or "not good."
At the same time, there is no straight forward way to monetize curation in order to pay the curator(s). Historically that has been via subscriptions to publications, but the resistance to paywalls and intrusive advertising (both mechanisms in use on today's network) have reduced the effectiveness of those techniques tremendously. Worse, the term "blog spam" is a pejorative for a reason, the blog owner, wrapping something they felt was worth reading in some prose and advertisements, do not compensate the author, and for many provide insufficient value to justify their attempts at monetization.
Getting back to the article's main point however, for specialized topics, finding that curation is difficult, getting paid to facilitate it more so. And that means that quality information (wisdom) is drowned in the sea of noise (hot takes).
My guess is that the Overton Window for paying for the work someone has done to distill the wisdom out of the signal has to move from the "radical" to the "acceptable" before we'll see much progress here.
Google any recipe you want to cook to experience this phenomenon.
If only we could apply collaborative filtering to webpages.
Low hanging wisdom: break long text into paragraphs!
Just wait til GPT-3 bots produce the information
It's already underway. I run into a handful of sites every week now. Here's an example https://testfoodkitchen.com/what-is-a-top-round-roast-good-f... - it seems plausible at first but the whole site is bizarre non-information.
From the site:
"Is top round tender?
Top round tender is a dish made from the top of a chicken’s head. The name comes from the fact that it is usually served at the peak of a chicken’s life, when its flesh is soft and pink. It is also called an “enormous” or “giardine” bird."
"Is top round tender?
Top round tender is a dish made from the top of a chicken’s head. The name comes from the fact that it is usually served at the peak of a chicken’s life, when its flesh is soft and pink. It is also called an “enormous” or “giardine” bird."
I just want to wait when they amass more shares and likes than nytimes.com and collectively have 100x as much content as the current sites.
They'll be the new Vice and Vox, but run entirely by AI
https://www.msn.com/en-za/news/other/ai-writing-is-here-and-...
They'll be the new Vice and Vox, but run entirely by AI
https://www.msn.com/en-za/news/other/ai-writing-is-here-and-...
Google search results vs reddit threads
- Reddit threads get pulled in search results.
- Reddit (and HN) are bad for "wisdom" because they are deliberately violating netiquette : once in a while, starting a new thread is wrong, and necroposting is right, and reddit prevents this by locking the thread.
- Reddit (and HN) are bad for "wisdom" because they are deliberately violating netiquette : once in a while, starting a new thread is wrong, and necroposting is right, and reddit prevents this by locking the thread.
Wiser words have rarely been spoken.
How is this article helping?
After that it always ends in hard to parse research papers, specifications, jumping around the source code of multiple libraries, debugging or reverse engineering. If I get lucky I might land on a 15 year old blog post that no man has seen since inception and it describes exactly what I wanted to know.
And that's just software. Humans know exactly what parts went into it and how it works. Why must wisdom always be squeezed out of a rock? Why is only the fluff copied thousands of times making research harder each day? Why are there never indicators which way to dig for more details?
T__T