11. Why the First “High Dimension” is Six or Maybe Five

(Epistemic status: morally correct, in the mathematician’s sense; here to give flavor and intuition without too much rigor.)

People toss around the concept of “high-dimensional space” a lot without having a good idea of where that starts. Interpretability researchers staring at vectors in hundreds of dimensions are clearly contemplating a high-dimensional space - loss landscapes packed full of saddle points and spheres where any given pair of vectors is almost certainly nearly orthogonal - but do they know what it means to be a high-dimensional space? Surely we can do better for ourselves than Justice Stewart’s infamous maxim - “I know it when I see it”!

What do we care about, when we want to tell how many orthogonal axes we need in a space to start calling it high-dimensional? I claim that we should care about looking at unit-radius disks - like, literal filled-in two-dimensional circles - and in particular, checking when, whether, and how pairs of those unit-radius disks overlap when we put their centers close together up to some small error (d << 1) and with arbitrarily chosen facings, that is, normal vectors to the disks. (My thesis here still holds up if we also have only small disagreements in facing, but it’s harder to see.) In particular, we should recognize a space as high-dimensional when such pairs of disks overlap in at most a single point - or better yet, fail to overlap at all - with probability ~1. There are a few reasons why we should operationalize high-dimensionality in this way.

First, when looking for unexpected structure in a point-cloud of arbitrary dimension, we might look for circles that don’t bound disks, and to a lesser extent, spheres that have hollow insides. This comes from topology: the kinds of loops that live within a space and can’t be pulled tight or otherwise smoothly shrunk to a point tell us a lot about the kind of space the loops live in - this lets us tell apart basketballs from inner tubes from solid donuts, for example. This is the foundation for what’s called the fundamental group of a space, which I’ll discuss in more detail in a future post. We care a little bit about how this extends to higher-dimensional circles, like spheres and hyperspheres, and the way that those can live in spaces and sometimes be hollow in complicated ways, but most of the interesting qualitative information lives at the very base of the hierarchy. Additionally, we might care about how rotation interacts with the dimensionality of a space, but again, rotations always live inside of a (2D) plane, so we’re right back to thinking about little segments of planes (that might be tangent to some hypersphere) and how they interact. Likewise, we want to go looking for when two such disks generally intersect in only a point, or not at all: this is what orthogonality looks like, as we’ll see in more detail. As an unashamed argument from authority - or if you prefer, “all the cool kids do it this way” - mathematical convention has it that high-dimensional topology starts at five or six dimensions, with three or four (and sometimes five) being low or middle dimensional, depending on your terminology, and two and fewer being so low as to be trivial; the operationalization I pick here recovers this definition. I claim that we know whereof we speak, having spent as many as several thousand seconds thinking about the topic; maybe more. Ultimately, though, I’m not going to back any of this up with fancy attempts at pictures or visualizations, or even a program to run the empirics; other people can do that if they want. I have intuition and I want to convey it to you. Believe me and accept the intuition or don’t. Let’s begin.

To reiterate, we’ll think about what happens when you look at the generic intersections of two unit disks whose centers are offset by some tiny distance - maybe one where the square of the ratio of that distance can comfortably be neglected - in some arbitrary direction, and whose facings are chosen arbitrarily - or, if you prefer, with some similarly small error such that the small angle approximation holds. The ambient space is high-dimensional if the resulting intersections are limited to a point, or ideally nothing at all.

First, the clearly low dimensions. A zero-dimensional space is too small to contain anything but a single point, never mind a single disk, let alone two! It’s low dimensional. Likewise for a one-dimensional space: too small to fit anything we care about. A two-dimensional space at least lets us fit disks in it, but the notionally arbitrary facings are actually limited to a single one: whatever the plane demands. As such, the overlap will always be something of a lens shape - two-dimensional. Much too large. Still low-dimensional.

Now for the low-but-interesting dimensions, that is, the middle dimensions. In a three-dimensional space, we finally get to think about the disks’ facings, which is promising! The overlap will always be a (1D) chord, though, unless we get extremely unlucky and the disks are parallel, giving us either a lens-like overlap again or no overlap at all, with an even more utterly vanishing chance of very weird tangency. We can ignore all of these rare cases: 3D is still not high-dimensional. For a four-dimensional space, the story can be a little bit harder to visualize, but it’s much the same, except a dimension lower: the overlap is nearly always a point. To see this, envision a 3D subspace and without loss of generality, fixing one of the disks as being centered on the origin with normal vector given by the z-axis. In the vanishingly rare cases where the other disk’s center is very close to the x-y plane, the nature of its facing gives us either part of a chord or no intersection at all, depending on how far its facing is from the z-axis. Otherwise, its intersection with the x-y-z slice looks like a line segment, nearly always meeting our first disk in a single point.

The fact that we generally end up with single-point intersections in four dimensions should give us a sense that we’re nearly done, and we are. In a five-dimensional space, the two disks only rarely intersect, and when they do, the analogue of the second disk in the 4D construction intersects the x-y-z slice in a single point, which must also be the point of intersection. In a six-dimensional space, the story is even more clear: that second disk will basically never intersect the x-y-z slice at all! We have arrived at the high dimensions; the first high dimension is six, or maybe five.

Another couple of ways to see that five is maybe the first high dimension and six surely is if not: since 5 = 2 + 2 + 1, five dimensions is the fewest where two planes can be skew, and thus also where double-rotations are generic. In a similar vein, if you stand at the origin in a 5D space and want to point your head to look at a star (or the sun!) in a 5D sky, you have four independent parameters for your facing, such that you have to align two planes’ worth of normal vector nearly perfectly to see it. And if that’s not enough to be quite high-dimensional, adding a single additional orthogonal direction surely is. The first high dimension is six, or maybe five, and all our low-dimensional geometric intuitions rapidly break down past that point: be very careful that you don’t still try to cling to them if your objects of study live in vastly higher-dimensional spaces.


14441/32768