An n-ball Between n-balls(arnaldur.be)
arnaldur.be
An n-ball Between n-balls
https://www.arnaldur.be/writing/about/an-n-ball-between-n-balls
59 comments
A good way to conceptualize what’s going on is not the idea that balls become "spiky" in high dimensions – like the article says, balls are always perfectly symmetrical by definition. But it’s the box becoming spiky, "caltrop-shaped", its vertices reaching farther and farther out from the origin as the square root of dimension, while the centers of its sides remain at exactly +-1. And the 2^N surrounding balls are also getting farther from the origin, while their radius remains 1/2. Now it should be quite easy to imagine how the center ball gets more and more room until it grows out of the spiky box.
But there is another way to think of the high-dimensional balls where "spikey" is the right visualisation.
Consider the volume of a cap. Take a plane that is 90% of the distance from the centre to the edge, and look at what percentage of the volume is "outside" that plane. When the dimension is high, that volume is negligible.
And when the dimension is really high you can get quite close to the centre, and still the volume you cut off is very small. In our 3D world the closest thing that has this property is a spike. You can cut off quite close to the centre, and the volume excised is small.
The sense in which a high-dimensional ball is not a spikey thing is in the symmetry, and the smoothness.
So when you want to develop an intuition for a high dimensional ball you need to think of it as simultaneously symmetrical, smooth, and spikey.
Then think of another five impossible things, and you can have breakfast.
Consider the volume of a cap. Take a plane that is 90% of the distance from the centre to the edge, and look at what percentage of the volume is "outside" that plane. When the dimension is high, that volume is negligible.
And when the dimension is really high you can get quite close to the centre, and still the volume you cut off is very small. In our 3D world the closest thing that has this property is a spike. You can cut off quite close to the centre, and the volume excised is small.
The sense in which a high-dimensional ball is not a spikey thing is in the symmetry, and the smoothness.
So when you want to develop an intuition for a high dimensional ball you need to think of it as simultaneously symmetrical, smooth, and spikey.
Then think of another five impossible things, and you can have breakfast.
I think our geometric intuition could never be good for a high dimensional object. Consider a sphere in 3D. It's represented by the points which satisfy x^2 + y^2 + z^2 = 1. Because there are only three coordinates, knowing the value of one of them greatly reduces the possibilities for the other two. For example, if I know z is close to 1, then my point is close to the north pole (x and y are both close to zero).
However, if I have the n-sphere x_1^2 + x_2^2 + ... + x_n^2 = 1, knowing the value of x_1 gives me very little information about all the other coordinates. And humans' interaction with the geometry is reality is somewhat limited to manipulating one coordinate at a time, i.e., our intuition is for built on things like moving our body linearly through space, not dilating the volume or surface area of our bodies.
However, if I have the n-sphere x_1^2 + x_2^2 + ... + x_n^2 = 1, knowing the value of x_1 gives me very little information about all the other coordinates. And humans' interaction with the geometry is reality is somewhat limited to manipulating one coordinate at a time, i.e., our intuition is for built on things like moving our body linearly through space, not dilating the volume or surface area of our bodies.
If x_1 is close to one, then the more x_n values there are the closer they are to zero.
That's a good point. High dimensional objects can obtain very unintuitive properties, like you describe.
This to me feels similar in many ways to how a corner in a high dimensional n-cube, although 90 degrees, no matter how you measure it, seems extremely spiky. As the shape does not increase in width, but the corners extend arbitrarily far away from the center. A property reserved for spiky things in 3D.
This to me feels similar in many ways to how a corner in a high dimensional n-cube, although 90 degrees, no matter how you measure it, seems extremely spiky. As the shape does not increase in width, but the corners extend arbitrarily far away from the center. A property reserved for spiky things in 3D.
Exactly: a corner of a square covers 1/4 of that part of the plane. A corner of a cube covers 1/8, a corner of a hypercube in dimension n covers just 1/(2^n) of the space. But each side/face/hyperface divides the plane/space/n-dim space just in half.
Ok, now i started to understand something. Thank you.
In some sense, a ball is more “natural” than a cube in Euclidean n-space. Once you add the metric, cubes become artificial constructs (despite being the natural elements in just product spaces).
TFA does say this too, right after the Hamming lecture.
"So instead of considering n-balls to be spiky, it’s the space around them that outgrows them."
"So instead of considering n-balls to be spiky, it’s the space around them that outgrows them."
Yes, but that doesn’t yet give me much intuition, so I wanted to elaborate on an idea I’ve found somewhat useful.
To be clear: the commenter you replied to just seems to be reiterating the idea, so I'm not accusing them of not reading the article.
You should have seen how few replies read the last article I posted. https://news.ycombinator.com/item?id=40525629 The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.
You should have seen how few replies read the last article I posted. https://news.ycombinator.com/item?id=40525629 The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.
> The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.
I have thoughts about this, but I can find no way to contact you to open a conversation. You might want to think about adding something to your HN profile.
Meanwhile ... from this and other sources, I feel your pain.
I have thoughts about this, but I can find no way to contact you to open a conversation. You might want to think about adding something to your HN profile.
Meanwhile ... from this and other sources, I feel your pain.
I read the article and the threads (to a point). It's hilarious that in the article you describe LLMs as being like hasty people jumping to conclusions, and the commenters on HN do exactly that thing in their comments.
Haha, that's a really funny similarity. I appreciate you sharing it :)
Very good to see this as the top-voted comment. I completely agree that this seems like a more natural explanation of what is going on.
even then the edges do not suddenly curve. its just all round a bad analogy.
This is a really good demonstration of the curse of dimensionality[0]
[0]: https://en.m.wikipedia.org/wiki/Curse_of_dimensionality
[0]: https://en.m.wikipedia.org/wiki/Curse_of_dimensionality
Interesting how this relates to LLMs scaling laws.
Why did I imagine that this would be about two shapes that are merely topologically n-balls, each having part of their boundary be incident with one of the two hemi(n-1)-spheres of the boundary of an n-ball (and otherwise not intersecting it)? (So like, in 3D, if you took some ball and two lumps of clay of different colors, and smooshed each piece of clay over half of the surface of the ball, with each of the two lumps of clay remaining topologically a 3-ball.)
I don’t know that there would even be anything interesting to say about that.
I don’t know that there would even be anything interesting to say about that.
I can't tell you why you imagined that, but that's pretty funny nevertheless.
Impressive, helpful, and now time to rebuild my own embeddings so I can grasp that red n-ball with my new n-D hands.
For other HN discussions of this phenomenon you can see some previous submissions of another article on it.
That article doesn't have the nice animations, but it is from 14 years ago ...
https://news.ycombinator.com/item?id=12998899
https://news.ycombinator.com/item?id=3995615
And from October 29, 2010:
https://news.ycombinator.com/item?id=1846682
That article doesn't have the nice animations, but it is from 14 years ago ...
https://news.ycombinator.com/item?id=12998899
https://news.ycombinator.com/item?id=3995615
And from October 29, 2010:
https://news.ycombinator.com/item?id=1846682
I also wrote a short blog post about this topic back in 2009: https://mark.reid.name/blog/warning-high-dimensions.html
It great to see these interesting math facts continue to be discussed and presented in new ways.
It great to see these interesting math facts continue to be discussed and presented in new ways.
I am struggling to juggle the balls in my mind. Are there any stepping-stone visual pieces like this to hopefully get me there? Very neat write-up, but I can't wait to share the realized absurdity of the red ball's green box eclipsing in our 3D intersection of the fully diagonalized 10D construct
The hypercube is the strange thing, not the red sphere. Placing the blue spheres tangent to the hypercube is an artificial construct which only “bounds” the red sphere in small dimensions. Our intuition is wrong because we think of the problem the wrong way (“the red sphere must be bounded by the box”, but there is no geometrical argument for that in n dimensions).
How does the enclosed sphere's radius changes with the number of dimensions, if the enclosing spheres are the following:
2D: 3 mutually touching 2-spheres (circles)
3D: 4 mutually touching 3-spheres (or spheres)
...
This variation of the problem doesn't rely on an artificial construct of a hypercube, I wonder if this yields a similarly unintuitive result.
2D: 3 mutually touching 2-spheres (circles)
3D: 4 mutually touching 3-spheres (or spheres)
...
This variation of the problem doesn't rely on an artificial construct of a hypercube, I wonder if this yields a similarly unintuitive result.
If my calculations are correct, then for this variation the enclosed n-sphere's radius converges to sqrt(2)-1 from below, and remains enclosed in the bounding hyper-tetrahedron.
Very interesting, I've considered doing something similar with other regular polyhedra, like the n-simplex (the one you analyzed) and n-orthoplex.
What was the side length in your calculation? did you find an equation for the size of the center n-ball?
What was the side length in your calculation? did you find an equation for the size of the center n-ball?
Buf, you may be right but I just cannot visualize it. It took me quite a while to do for the cube, imagine a tetrahedron. But you might be right.
Can I just say how my mind is utterly blown by the animations
Thank you <3
The trigonometry involved was pretty intense at times.
Numberphile did a video on this a while back.
https://youtu.be/mceaM2_zQd8?si=0xcOAoF-Bn1Z8nrO
[deleted]
wow discovering Hamming’s lecture was enough for me! so good
The whole series is on Youtube, awesome.
The course was also the basis for his book _The Art of Doing Science and Engineering_ (1997). At first it takes some getting used to as you have the feeling it may be outdated, but it's about teaching a style of thinking. It's great.
The course was also the basis for his book _The Art of Doing Science and Engineering_ (1997). At first it takes some getting used to as you have the feeling it may be outdated, but it's about teaching a style of thinking. It's great.
I know, I didn't realize he was alive in the 90s! Hearing him (sarcastically) say "now having 10 parameters isn't unusual, correct?" makes me wish he could've seen the 60B-parameter curve fitting we're doing nowadays.
The whole lecture series was about giving students a style of thinking that might hopefully prepare them for the future, without focusing on special knowledge (there were other courses for that).
Two of the lectures were spent on building intuition for very high dimensionality (this one), and another on neural networks, because he thought there was a big chance they were going to be important. In the early 90s, not bad.
Two of the lectures were spent on building intuition for very high dimensionality (this one), and another on neural networks, because he thought there was a big chance they were going to be important. In the early 90s, not bad.
I think he already knew something about it... it talks about AI. Maybe at the time 100k to 1M dimensions? A bright mind like his, could very good extrapolate to 2024.
Yes sir! I've just bookmarked the series... Priceless!
Both ChatGpt 4.o and Claude failed to answer
“…At what dimension would the red ball extend outside the box?”
If anyone has o1-preview it’d be interesting to hear how well it does on this.
“…At what dimension would the red ball extend outside the box?”
If anyone has o1-preview it’d be interesting to hear how well it does on this.
This was the prompt I gave o1-preview:
> There is a geometric thought experiment that is often used to demonstrate the counterintuitive shape of high-dimensional phenomena. We start with a 4×4 square. There are four blue circles, with a radius of one, packed into the box. One in each corner. At the center of the box is a red circle. The red circle is as large as it can be, without overlapping the blue circles. When extending the construct to 3D, many things happen. All the circles are now spheres, the red sphere is larger while the blue spheres aren’t, and there are eight spheres while there were only four circles.
> There are more than one way to extend the construct into higher dimensions, so to make it more rigorous, we will define it like so: An n-dimensional version of the construct consists of an n-cube with a side length of 4. On the midpoint between each vertex and the center of the n-cube, there is an n-ball with a radius of one. In the center of the n-cube there is the largest n-ball that does not intersect any other n-ball.
> At what dimension would the red ball extend outside the box?
Response: "[...] Conclusion: The red ball extends outside the cube when n≥10n≥10."
It calculated it with a step-by-step explanation. This is the first time I'm actually pretty stunned. It analysed the problem, created an outline. Pretty crazy.
> There is a geometric thought experiment that is often used to demonstrate the counterintuitive shape of high-dimensional phenomena. We start with a 4×4 square. There are four blue circles, with a radius of one, packed into the box. One in each corner. At the center of the box is a red circle. The red circle is as large as it can be, without overlapping the blue circles. When extending the construct to 3D, many things happen. All the circles are now spheres, the red sphere is larger while the blue spheres aren’t, and there are eight spheres while there were only four circles.
> There are more than one way to extend the construct into higher dimensions, so to make it more rigorous, we will define it like so: An n-dimensional version of the construct consists of an n-cube with a side length of 4. On the midpoint between each vertex and the center of the n-cube, there is an n-ball with a radius of one. In the center of the n-cube there is the largest n-ball that does not intersect any other n-ball.
> At what dimension would the red ball extend outside the box?
Response: "[...] Conclusion: The red ball extends outside the cube when n≥10n≥10."
It calculated it with a step-by-step explanation. This is the first time I'm actually pretty stunned. It analysed the problem, created an outline. Pretty crazy.
I never understood the need to distinguish between "ball" and "sphere" in maths. Sure, one is solid and the other hollow, but why is that fact so important that you need to use a completely different word? As I understand it, you could replace every instance of "ball" in this article with "sphere" and it would still be correct.
We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?
We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?
> you could replace every instance of "ball" in this article with "sphere" and it would still be correct.
I completely agree. N-ball is maybe more mathematically precise. It might rule out some strange edge case I couldn't think of. I chose it mostly for stylistic reasons. n-ball is shorter than n-sphere, and ball is a more playful term.
I do however understand the need to define the two object classes, and see little issue with giving them distinct names.
I completely agree. N-ball is maybe more mathematically precise. It might rule out some strange edge case I couldn't think of. I chose it mostly for stylistic reasons. n-ball is shorter than n-sphere, and ball is a more playful term.
I do however understand the need to define the two object classes, and see little issue with giving them distinct names.
[deleted]
> We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?
Topologists don't need them because they already have ball and sphere.
In analysis, I can imagine them calling full hyperrectangles "brick", and empty hyperrectangles "box", but both words start with "b", so there is no shorthand for them on paper. I^n and ∂I^n are just fine.
Topologists don't need them because they already have ball and sphere.
In analysis, I can imagine them calling full hyperrectangles "brick", and empty hyperrectangles "box", but both words start with "b", so there is no shorthand for them on paper. I^n and ∂I^n are just fine.
And, indeed, it’s probably in topology where the distinction between sphere and ball is most important.
We do for 2D though - Circle and disc. And in 4D plus we have the 3-sphere and 4-ball, etc.
And for 3D shapes have Torus, and, well, ‘solid-torus’.
And for 3D shapes have Torus, and, well, ‘solid-torus’.
I… can’t.
Matlock, is that you?
I am not Matlock, who is that?
He probably meant MacGyver.
https://en.m.wikipedia.org/wiki/MacGyver
https://en.m.wikipedia.org/wiki/MacGyver
No, he really did mean Matlock. Grey hair guy in a white suit lecturing...Fictional Lawyer.
https://www.alamy.com/stock-photo/matlock-tv-andy-griffith.h...
Reboot on TV this year.
Reboot on TV this year.
I know who Matlock is. But then it makes no sense at all.