3D interfaces are usually worse than 2D interfaces(facebook.com)
facebook.com
3D interfaces are usually worse than 2D interfaces
https://www.facebook.com/permalink.php?story_fbid=2407256322842204&id=100006735798590
117 comments
I think it's more than that. As Carmack observes, you don't see in 3D. You see two 2D planes that your brain extracts a certain amount of depth information from. As a UI goes, you can't actually freely use that third dimension, because as soon as one element obscures another, either the front element is too opaque to see through, in which case the second might as well not be there, or the opacity is not 100% in which case it just gets confusing fast.
You've really got something more like 2.1d to work with with humans; intefaces need to take that into account, and maybe sometimes use that .1d depth dimension to provide light hinting, but not depend on it.
So you're not removing a dimension... you're acknowledging it doesn't exist. There isn't really a "third dimension" here to take advantage of. It may be visually impressive if something flies from far away to up close, but in interface terms, all that has happened is that the z-buffer values shifted, but the interface still has the same number of square radians available that it did before.
The depth information is a lot more like an additional color than an additional spatial dimension. UIs are better with color hinting available when used well, but there's not much fundamental difference between a color 2D interface and a monochrome 2D interface. Proof that it isn't much is that there are some people who use settings to set their phone to monochrome, and the result is at worst occasionally inconvenient, not crippling. Compare with trying to go from a 2D interface to a 1D interface, which would be utterly crippling and destroy a huge swathe of current UI paradigms.
To truly "see in 3D" would require a fourth-dimension perspective. A 4D person could use a 3D display arbitrarily, because they can freely see the entire 3D space, including seeing things inside opaque spheres, etc, just like we can look at a 2D display and see the inside of circles and boxes freely. They have access to all sorts of paradigms we don't, just like no 2D designer would ever come up with a "text box", even though it fits on the 2D plane, because no 2D user could see inside the text box to see what's in it.
(This is one of those cases where the mere act of typing it out improved my understanding of the topic. I didn't have the "depth information is really just another color" idea going in, but the more I think about it the more sense it makes, in terms of the amount of information that channel can carry. Just like color, it's not quite 0, but you can't really stuff all that much in there.)
You've really got something more like 2.1d to work with with humans; intefaces need to take that into account, and maybe sometimes use that .1d depth dimension to provide light hinting, but not depend on it.
So you're not removing a dimension... you're acknowledging it doesn't exist. There isn't really a "third dimension" here to take advantage of. It may be visually impressive if something flies from far away to up close, but in interface terms, all that has happened is that the z-buffer values shifted, but the interface still has the same number of square radians available that it did before.
The depth information is a lot more like an additional color than an additional spatial dimension. UIs are better with color hinting available when used well, but there's not much fundamental difference between a color 2D interface and a monochrome 2D interface. Proof that it isn't much is that there are some people who use settings to set their phone to monochrome, and the result is at worst occasionally inconvenient, not crippling. Compare with trying to go from a 2D interface to a 1D interface, which would be utterly crippling and destroy a huge swathe of current UI paradigms.
To truly "see in 3D" would require a fourth-dimension perspective. A 4D person could use a 3D display arbitrarily, because they can freely see the entire 3D space, including seeing things inside opaque spheres, etc, just like we can look at a 2D display and see the inside of circles and boxes freely. They have access to all sorts of paradigms we don't, just like no 2D designer would ever come up with a "text box", even though it fits on the 2D plane, because no 2D user could see inside the text box to see what's in it.
(This is one of those cases where the mere act of typing it out improved my understanding of the topic. I didn't have the "depth information is really just another color" idea going in, but the more I think about it the more sense it makes, in terms of the amount of information that channel can carry. Just like color, it's not quite 0, but you can't really stuff all that much in there.)
True, almost any interface that overlaps helpful data in layers or stacks is usually terrible. Hell, I don't even like desktop computer GUIs that allow you to have stacks of windows. I'd rather see one thing at a time and cycle between them. Or have a picker view.
That said we do actually get quite a bit out of that ability to see depth. People who lose depth perception have quite a hard time adapting to the world. And our spatial understanding seems to go beyond our vision. That to me is where a 3D interface might be really powerful. Sometimes things which we struggle to decode in 2D are just intuitive in 3D like knots or the run of wires or pipes.
As I said elsewhere in this thread I think the 3D interfaces that are really going to be powerful haven't occurred to us yet. And I believe that what we'll find in time is that there are things which 3D interfaces are tremendously advantagous for and using anything else will feel like a hinderance. But those will be things for which 2D interfaces don't already do an amazing job.
That said we do actually get quite a bit out of that ability to see depth. People who lose depth perception have quite a hard time adapting to the world. And our spatial understanding seems to go beyond our vision. That to me is where a 3D interface might be really powerful. Sometimes things which we struggle to decode in 2D are just intuitive in 3D like knots or the run of wires or pipes.
As I said elsewhere in this thread I think the 3D interfaces that are really going to be powerful haven't occurred to us yet. And I believe that what we'll find in time is that there are things which 3D interfaces are tremendously advantagous for and using anything else will feel like a hinderance. But those will be things for which 2D interfaces don't already do an amazing job.
Carmack already mentioned the existence of "true 3D" content, for which you get a 3D interface whether you like it or not, so to speak, so I didn't go into that.
But making everything 3D, because VR, is as silly as when the gaming industry made everything 3D, resulting in entire console libraries full of games that looked like shit even at the time, pushing 4 or 5 frames per second and having other incredible compromises, when the same consoles are monsters of 2D performance. As nice as it may be to have truly 3D content available in psuedo-real space, there's no reason to insist that when you want to set the shininess of a given pipe that you need a huge skueomorphic switch as big as an old car stick shift that you can visibly pull popping out of your UI or something when all you need is a slider. (If anything, I'd think minimalism in a VR environment is a good idea, both to contrast the other content and to prevent detracting from it.)
I think that's probably the kind of crap Carmack is complaining about. We've already been around the same loop a couple of times already, and 3D, albeit on 2D surfaces, was one of them, so it's fair to look to the past instances of such BS and maybe this time try to move along the curve a bit faster. I'd say that if we can get this nonsense out of the way faster rather than slower, we're more likely to get to the truly useful 3D stuff that doesn't exist yet. Otherwise we risk 3D interfaces becoming something like the Wiimote, which IMHO was actually a really useful tool that has become despised solely because it was badly misused by so many games, because motion controls. (Another example we've already been through.)
But making everything 3D, because VR, is as silly as when the gaming industry made everything 3D, resulting in entire console libraries full of games that looked like shit even at the time, pushing 4 or 5 frames per second and having other incredible compromises, when the same consoles are monsters of 2D performance. As nice as it may be to have truly 3D content available in psuedo-real space, there's no reason to insist that when you want to set the shininess of a given pipe that you need a huge skueomorphic switch as big as an old car stick shift that you can visibly pull popping out of your UI or something when all you need is a slider. (If anything, I'd think minimalism in a VR environment is a good idea, both to contrast the other content and to prevent detracting from it.)
I think that's probably the kind of crap Carmack is complaining about. We've already been around the same loop a couple of times already, and 3D, albeit on 2D surfaces, was one of them, so it's fair to look to the past instances of such BS and maybe this time try to move along the curve a bit faster. I'd say that if we can get this nonsense out of the way faster rather than slower, we're more likely to get to the truly useful 3D stuff that doesn't exist yet. Otherwise we risk 3D interfaces becoming something like the Wiimote, which IMHO was actually a really useful tool that has become despised solely because it was badly misused by so many games, because motion controls. (Another example we've already been through.)
Your points are excellent, classic.
One other hypothetical use of a 3D interface is as a way to conceptualize "true N-dimensional" data. A 3D experience indeed doesn't help any rational conceptualize of a rational situation but it might, maybe, allow you to mobilize the unconscious reflexes humans have for dealing with regular 3d space. But all this might also a 90s cyberpunk fantasy.
One other hypothetical use of a 3D interface is as a way to conceptualize "true N-dimensional" data. A 3D experience indeed doesn't help any rational conceptualize of a rational situation but it might, maybe, allow you to mobilize the unconscious reflexes humans have for dealing with regular 3d space. But all this might also a 90s cyberpunk fantasy.
[deleted]
I think what you've said is all correct. And its mimicked, to some degree, by how we build our physical lives. Shelves and desks are not so deep as to obscure what you need.
That said, I think an element that's missed is tangibility. In the real world, a stack of papers and folders are a valid way organize your own desk. You know where everything is and allowing you to tune your layout and grow muscle memory is very helpful. VR interfaces should let the user organize as they see fit. Interfaces should be tangible, movable, sizable, etc.
That said, I think an element that's missed is tangibility. In the real world, a stack of papers and folders are a valid way organize your own desk. You know where everything is and allowing you to tune your layout and grow muscle memory is very helpful. VR interfaces should let the user organize as they see fit. Interfaces should be tangible, movable, sizable, etc.
> And its mimicked, to some degree, by how we build our physical lives. Shelves and desks are not so deep as to obscure what you need.
Oddly, cupboards and refrigerators are. A lot of food rots in the back of refrigerators that should have just been smaller.
Oddly, cupboards and refrigerators are. A lot of food rots in the back of refrigerators that should have just been smaller.
Maybe someone should build a cylindrical fridge with a curved door, so you could rotate the shelves to get to things diametrically across from your view.
It needn’t have structural pole in the middle. Could use roller bearings along the perimeter.
I think it would work better than an ultra-wide but shallow fridge.
It needn’t have structural pole in the middle. Could use roller bearings along the perimeter.
I think it would work better than an ultra-wide but shallow fridge.
Fridge depth is useful for storing large objects like a 24 pack of water. The trick is keeping them mostly empty otherwise food will rot before you eat it.
Bizarrely, counter-depth (aka depth that both looks better in kitchens and is shallow enough you won't forget about half the food in it) fridges are usually a fair bit more expensive, despite being smaller. Even for what's otherwise the same model.
I agree with your analysis, but seeing 3D is probably not the most important aspect of a potential useful 3D interface.
The human ability to intuitively internally model and navigate 3D spaces is what makes 3D special.
Cf. method of loci, for instance.
The human ability to intuitively internally model and navigate 3D spaces is what makes 3D special.
Cf. method of loci, for instance.
> the interface still has the same number of square radians available that it did before
This is really well put. And your "UI depth = colour" is good too, it really is more just hinting than an actual data channel.
This is really well put. And your "UI depth = colour" is good too, it really is more just hinting than an actual data channel.
yes we see 2d surfaces in 3d. Dolphins can actually see in 3d as they can sense entire 3d volumes and project them back to eachother too.
I’m assuming you meant “yes we see 3d surfaces in 2d” to be consistent with the parent comment? Also, I’m impressed that dolphins have that ability. How does projecting them to each other work biologically? Do they somehow encode what they see as sound (less lossily than humans do)?
I'd imagine they work directly in the frequency domain. They don't "see with sound", they'll just hear nested spheres or whatever.
> The usefulness of most interfaces is that they abstract something real.
When I think someone is overenthusiastic about virtual reality applications (especially stuff involving "collaboration") I like to contrast it with an insanely-successful tool with broad adoption: E-mail.
Who wants to take their message, drop it into a virtual envelope, affix a virtually-licked stamp and virtually-walk to the local virtual-post-office?
Heck no, we just click the big "Send" button! We didn't model the cruft in 3D, we removed it, because it wasn't truly necessary to the job.
When I think someone is overenthusiastic about virtual reality applications (especially stuff involving "collaboration") I like to contrast it with an insanely-successful tool with broad adoption: E-mail.
Who wants to take their message, drop it into a virtual envelope, affix a virtually-licked stamp and virtually-walk to the local virtual-post-office?
Heck no, we just click the big "Send" button! We didn't model the cruft in 3D, we removed it, because it wasn't truly necessary to the job.
On the one hand, I totally agree with this as my experience with AR/VR interfaces as well.
However, I'm not convinced that this is due to any inherent limitations in our brain's ability to understand a third dimension, just that we are accustomed to only having to deal with two dimensions. I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus.
For example, in many spaceflight simulators, the radar includes vertical bars to indicate height above/below your plane. That's a 3-D interface rendered onto a 2-D screen, though it's 3-D in VR. Once you learn to read it, you become highly effective at mapping what you see to a spatial understanding of what's going on around you.
The driver's controls in a car are also a good example of a 3-D interface demonstrating patterns that are inherently 3-D (twist knobs, paddles, H pattern gearbox, etc.), and it works mostly because you are expected to develop muscle memory.
However, I'm not convinced that this is due to any inherent limitations in our brain's ability to understand a third dimension, just that we are accustomed to only having to deal with two dimensions. I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus.
For example, in many spaceflight simulators, the radar includes vertical bars to indicate height above/below your plane. That's a 3-D interface rendered onto a 2-D screen, though it's 3-D in VR. Once you learn to read it, you become highly effective at mapping what you see to a spatial understanding of what's going on around you.
The driver's controls in a car are also a good example of a 3-D interface demonstrating patterns that are inherently 3-D (twist knobs, paddles, H pattern gearbox, etc.), and it works mostly because you are expected to develop muscle memory.
Are car controls really a good example of a 3D interface in the sense Carmack is discussing? I think he's specifically talking about 3D visualization.
In contrast car interfaces are specifically non-visual. You have to look to learn how to use them, but after that, anything you have to look at to use is bad, because it takes the driver's eyes off the road. But in the same way that the human eyes extract a 2D version of a 3D world, I think the a car's nobs are a proprioceptive projection of the 3D world.
In contrast car interfaces are specifically non-visual. You have to look to learn how to use them, but after that, anything you have to look at to use is bad, because it takes the driver's eyes off the road. But in the same way that the human eyes extract a 2D version of a 3D world, I think the a car's nobs are a proprioceptive projection of the 3D world.
I know Carmack was specifically talking about 3D visualization, but I was responding to the parent comment's point about the value of data simplification that 2D brings. I don't usually quote myself, but that's what I meant by:
I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus.
I specifically brought up the car's knobs and wheels and buttons because it's an example of a fairly complex but widely adopted and successful non-2D interface. My point was that I don't think there is an inherent limitation in the human brain's ability to comprehend or think in 3D, and that I think it's more of a training and acclimatization phenomenon.
I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus.
I specifically brought up the car's knobs and wheels and buttons because it's an example of a fairly complex but widely adopted and successful non-2D interface. My point was that I don't think there is an inherent limitation in the human brain's ability to comprehend or think in 3D, and that I think it's more of a training and acclimatization phenomenon.
And I'm saying I think it's not actually the brain thinking in 3D. I think it's thinking in terms of body position and movement. Which is something that takes place in a 3D space, but isn't actually a generalized 3D comprehension.
One way to check would be to have people reproduce things they know via touch in visual contexts. E.g., can people draw their steering wheel? Given how their tongue knows their teeth, can they do a 3D model of that without looking at references? If we hand people a box where they can put their hand in but can't look, can they model the shape just as well as they could by looking at it?
For me at least, these are very different kinds of knowledge. If I'm, say, feeling around and working on the back of a server I can't see, actually looking at it is a very different experience.
One way to check would be to have people reproduce things they know via touch in visual contexts. E.g., can people draw their steering wheel? Given how their tongue knows their teeth, can they do a 3D model of that without looking at references? If we hand people a box where they can put their hand in but can't look, can they model the shape just as well as they could by looking at it?
For me at least, these are very different kinds of knowledge. If I'm, say, feeling around and working on the back of a server I can't see, actually looking at it is a very different experience.
I would wager that unless you were a practiced artist/designer, you would have trouble replicating a 2d UI from memory with much accuracy. Our brains don't work like that in 2D either. You could probably tell me that ABC is in one corner and XYZ is in another, but you probably couldn't tell me the relative font sizes and widths and heights of boxes and cards. Unless you're a frontend developer or a UI designer, in which case you're trained to see and remember those things.
There are plenty of examples of people who are able to fluidly navigate 3D systems. RC helicopter pilots are a great example. The helicopter's controls are relative to it, not you, and it takes a while to retrain your brain to think in those terms. But you can see plenty of videos of stunt pilots pulling off tricks that clearly require an excellent understanding of the controls, the mechanics, and the space around the helicopter.
Sculptors are also an excellent example. Even with a reference, I'd bet that it would take you a really long time to replicate an object with clay. However, I once watched an artist friend carve a block of butter into a Buddha over a one hour dinner. I think that would pass your test for having an effective 3D mental model.
Speaking not as any authority on the topic, I'm arguing that our brains aren't usually asked to deal with 3d problems, so we don't have developed mental muscles for it. Except for the people who do deal with 3D regularly.
There are plenty of examples of people who are able to fluidly navigate 3D systems. RC helicopter pilots are a great example. The helicopter's controls are relative to it, not you, and it takes a while to retrain your brain to think in those terms. But you can see plenty of videos of stunt pilots pulling off tricks that clearly require an excellent understanding of the controls, the mechanics, and the space around the helicopter.
Sculptors are also an excellent example. Even with a reference, I'd bet that it would take you a really long time to replicate an object with clay. However, I once watched an artist friend carve a block of butter into a Buddha over a one hour dinner. I think that would pass your test for having an effective 3D mental model.
Speaking not as any authority on the topic, I'm arguing that our brains aren't usually asked to deal with 3d problems, so we don't have developed mental muscles for it. Except for the people who do deal with 3D regularly.
And I'm saying I think you're overgeneralizing. I don't think there is general 3D capability to be developed. E.g., I doubt a sculptor (who takes years to learn to do it well) swap chairs with a helicopter pilot (ditto) and have them both do reasonably well.
I of course agree people can learn to develop all sorts of specific skills. But I don't think specific skills that operate in 3 dimensions are an argument for VR being particularly useful for a general audience.
I of course agree people can learn to develop all sorts of specific skills. But I don't think specific skills that operate in 3 dimensions are an argument for VR being particularly useful for a general audience.
We seem to be talking past each other.
I'm saying there is no general 2D capability either, so the existence of a general 3D capability (which I doubt it exists) is irrelevant to the question.
And that is also separate from whether VR is useful for a general audience. VR becomes useful for a general audience the same way a smartphone or a computer does: by covering enough usages that the vast majority of users will find useful applications. That's regardless of whether those applications use a 2D UI projected into the virtual environment or a skeuomorphic 3D UI.
I'm saying there is no general 2D capability either, so the existence of a general 3D capability (which I doubt it exists) is irrelevant to the question.
And that is also separate from whether VR is useful for a general audience. VR becomes useful for a general audience the same way a smartphone or a computer does: by covering enough usages that the vast majority of users will find useful applications. That's regardless of whether those applications use a 2D UI projected into the virtual environment or a skeuomorphic 3D UI.
“twist knobs” – 1 degree of freedom
“paddles” – more or less a discrete switch, in form of a stick
“H pattern gearbox” – a few discrete choices
pedals – 1 degree of freedom
steering wheel – 1 degree of freedom
These things all exist in a 3-dimensional world, but they are not “3-D interfaces”.
Compare with https://en.wikipedia.org/wiki/SpaceOrb_360
* * *
If you define a car’s interface to be “3D” then so is basically anything, say a TV remote or the knobs on an oven.
“paddles” – more or less a discrete switch, in form of a stick
“H pattern gearbox” – a few discrete choices
pedals – 1 degree of freedom
steering wheel – 1 degree of freedom
These things all exist in a 3-dimensional world, but they are not “3-D interfaces”.
Compare with https://en.wikipedia.org/wiki/SpaceOrb_360
* * *
If you define a car’s interface to be “3D” then so is basically anything, say a TV remote or the knobs on an oven.
These are tactile controls arranged in a 3D space.
The 2D equivalent would be a flat screen with virtual knobs, wheels, and switches, and would be completely unusable.
The problem with AR is that it's 2.5D. Physical 3D is fully tactile in every dimension. AR lacks that kind of tactility.
True - not just visual - VR would have fingertip touch control of every object in the scene. That would (literally) be a game-changer, but we don't have the technology to make it happen yet.
The 3D "mice" that exist for AR/VR are clunky and crude with hand-level resolution rather than fingertip resolution.
Compare with a 2D touchscreen which has good tactile control. And a 2D desktop which (usually) gives you a bigger screen area in return for simplified but still very usable tactile control with a mouse.
In other words, VR/AR is not just about the visuals.
The 2D equivalent would be a flat screen with virtual knobs, wheels, and switches, and would be completely unusable.
The problem with AR is that it's 2.5D. Physical 3D is fully tactile in every dimension. AR lacks that kind of tactility.
True - not just visual - VR would have fingertip touch control of every object in the scene. That would (literally) be a game-changer, but we don't have the technology to make it happen yet.
The 3D "mice" that exist for AR/VR are clunky and crude with hand-level resolution rather than fingertip resolution.
Compare with a 2D touchscreen which has good tactile control. And a 2D desktop which (usually) gives you a bigger screen area in return for simplified but still very usable tactile control with a mouse.
In other words, VR/AR is not just about the visuals.
> These are tactile controls arranged in a 3D space.
They're arranged roughly on the same plane (the XY plane).
They're arranged roughly on the same plane (the XY plane).
> The driver's controls in a car are also a good example of a 3-D interface demonstrating patterns that are inherently 3-D (twist knobs, paddles, H pattern gearbox, etc.)
These are almost purely input mechanisms, in that sense we've had 3D interfaces in video games for decades by way various controller shapes. The problem is output interfaces don't work or at least don't gain much from the third dimension.
These are almost purely input mechanisms, in that sense we've had 3D interfaces in video games for decades by way various controller shapes. The problem is output interfaces don't work or at least don't gain much from the third dimension.
> H pattern gearbox
What's 3-D about that? The shifter moves in a two-dimensional pattern, though conceptually you're only moving in one dimension (up and down the gears).
I suppose you could treat the clutch pedal as the third dimension together with two dimensions for the H.
Knobs and paddles are likewise only single dimensional, though embedded in 3D space, from my perspective.
What's 3-D about that? The shifter moves in a two-dimensional pattern, though conceptually you're only moving in one dimension (up and down the gears).
I suppose you could treat the clutch pedal as the third dimension together with two dimensions for the H.
Knobs and paddles are likewise only single dimensional, though embedded in 3D space, from my perspective.
The controls exist on many different planes, and their placement requires some muscle memory, such as moving your hand between the steering wheel and the shifter. Each individual interaction in the interface might be constrained to two degrees of freedom, but the interface requires operating on a spatial mental model.
Maps are one very good example of this. A map is partially so powerful because it removes a ton of information you probably don't care about...Sure. But most of the time who cares.
This is usually true, but then again, there is the flip side. There are exceptions to this, and those exceptions may be interesting areas of inquiry. In particular, there are situations where one would like to go from a map representation to seeing relationships which can't be readily expressed in a map. One example of this would be evaluating views from a house. Another example, would be the layout of ducting within the ceiling or within crawl spaces. Another example would be evaluating lines of sight or lines of fire within a particular scene.
Often, this can be handled as something like a relief map or a scale model which can be manipulated and zoomed into by the user in VR, but the exceptions to the "a map is usually enough" seem to coincide with areas where VR/AR might be particularly interesting.
I think there might be some relationship here to Archy.
https://en.wikipedia.org/wiki/Archy
This is usually true, but then again, there is the flip side. There are exceptions to this, and those exceptions may be interesting areas of inquiry. In particular, there are situations where one would like to go from a map representation to seeing relationships which can't be readily expressed in a map. One example of this would be evaluating views from a house. Another example, would be the layout of ducting within the ceiling or within crawl spaces. Another example would be evaluating lines of sight or lines of fire within a particular scene.
Often, this can be handled as something like a relief map or a scale model which can be manipulated and zoomed into by the user in VR, but the exceptions to the "a map is usually enough" seem to coincide with areas where VR/AR might be particularly interesting.
I think there might be some relationship here to Archy.
https://en.wikipedia.org/wiki/Archy
I completely agree. A lot of what my team works on is finding those cases where a 3D view has a very high value. One example is a project we worked on that allows construction companies to use the 3D CAD data for their project as a setting to move equipment through. It's not unheard of for a company to bring a piece of equipment on site only to find it cannot move through a given space because material is in the way. Sometimes a partially complete structure which could've been built later is impassable. This is a situation in which a completely 3D representation is really powerful and the effort can save significant money in the long run.
I think there are absolutely cases where 3D interfaces will blow away 2D ones. We're on the road to discovering those. But the industry has had a tendency to try to take what works in 2D and make a 3D version, often ending with a user experience that is less intuitive and less powerful.
The 3D interfaces that are really going to make a difference will probably take decades to invent/discover. Our 2D interfaces have had literally thousands of years to develop.
I think there are absolutely cases where 3D interfaces will blow away 2D ones. We're on the road to discovering those. But the industry has had a tendency to try to take what works in 2D and make a 3D version, often ending with a user experience that is less intuitive and less powerful.
The 3D interfaces that are really going to make a difference will probably take decades to invent/discover. Our 2D interfaces have had literally thousands of years to develop.
It's not unheard of for a company to bring a piece of equipment on site only to find it cannot move through a given space because material is in the way.
This is precisely the kind of situation I was thinking about!
This is a situation in which a completely 3D representation is really powerful and the effort can save significant money in the long run.
I think this is a key criteria for determining what problems should be looked at.
This is precisely the kind of situation I was thinking about!
This is a situation in which a completely 3D representation is really powerful and the effort can save significant money in the long run.
I think this is a key criteria for determining what problems should be looked at.
Didn't read Raskin's book, but what's described in the "Commands" section on the Wiki page seems exactly like what UIs in (some) Lisp Machines worked, and what CLIM[0][1] does.
--
[0] - https://en.wikipedia.org/wiki/Common_Lisp_Interface_Manager
[1] - https://common-lisp.net/project/mcclim/
--
[0] - https://en.wikipedia.org/wiki/Common_Lisp_Interface_Manager
[1] - https://common-lisp.net/project/mcclim/
I strongly agree with your point about abstraction. However I'd love if people who work on such interfaces would get a stronger grip on how the perception of depth works.
Painters for example use saturation, colors, occlusion, details, abstraction and perspective to get the most out of depth.
In a 3D scene with the same objects you can make or break the scene's readability, purely based on the placement of objects. What you can't do, is to come and slap 3D onto it and it will somehow work out. If you think 3D space itself will make it magically work, you are wrong. Like a graphic designer puts a lot of work into making 2D information readable, 3D information also would need a ton of work to become readable.
Interestingly enough a lot of 2D designs actually emulate levels of depth by using occlusion, focus, shadows, etc.
I can certainly imagine 3D interfaces that would work, you would only have to be very disciplined about your use of dimensions and how much to have on which plane and why.
Painters for example use saturation, colors, occlusion, details, abstraction and perspective to get the most out of depth.
In a 3D scene with the same objects you can make or break the scene's readability, purely based on the placement of objects. What you can't do, is to come and slap 3D onto it and it will somehow work out. If you think 3D space itself will make it magically work, you are wrong. Like a graphic designer puts a lot of work into making 2D information readable, 3D information also would need a ton of work to become readable.
Interestingly enough a lot of 2D designs actually emulate levels of depth by using occlusion, focus, shadows, etc.
I can certainly imagine 3D interfaces that would work, you would only have to be very disciplined about your use of dimensions and how much to have on which plane and why.
Agree. the really great 3D interfaces we create will be 3D first and not a clumsy adaption of 2D ones.
I completely disagree with your maps example here. Yes, 2D currently works ok, but I can't see how it can beat a full-surround AR experience, which simply overlays a shiny string along the path you have to follow.
That is for real-time navigation.
For positioning and planning, 3D will be required to navigate a multistory mall. But a 3D map movie-style could potentially be replaced with something better, if researched.
That is for real-time navigation.
For positioning and planning, 3D will be required to navigate a multistory mall. But a 3D map movie-style could potentially be replaced with something better, if researched.
I recently posted some comments by Dave Ackley about 2D versus 3D, which I'll repeat here:
David Ackley, who developed the two-dimensional CA-like "Moveable Feast Machine" architecture for "Robust First Computing", touched on moving from 2D to 3D in his retirement talk:
https://youtu.be/YtzKgTxtVH8?t=3780
"Well 3D is the number one question. And my answer is, depending on what mood I'm in, we need to crawl before we fly."
"Or I say, I need to actually preserve one dimension to build the thing and fix it. Imagine if you had a three-dimensional computer, how you can actually fix something in the middle of it? It's going to be a bit of a challenge."
"So fundamentally, I'm just keeping the third dimension in my back pocket, to do other engineering. I think it would be relatively easy to imaging taking a 2D model like this, and having a finite number of layers of it, sort of a 2.1D model, where there would be a little local communication up and down, and then it was indefinitely scalable in two dimensions."
"And I think that might in fact be quite powerful. Beyond that you think about things like what about wrap-around torus connectivity rooowaaah, non-euclidian dwooraaah, aaah uuh, they say you can do that if you want, but you have to respect indefinite scalability. Our world is 3D, and you can make little tricks to make toruses embedded in a thing, but it has other consequences."
Here's more stuff about the Moveable Feast Machine:
https://news.ycombinator.com/item?id=15560845
https://news.ycombinator.com/item?id=14236973
The most amazing mind blowing demo is Robust-first Computing: Distributed City Generation:
https://www.youtube.com/watch?v=XkSXERxucPc
And a paper about how that works:
https://www.cs.unm.edu/~ackley/papers/paper_tsmall1_11_24.pd....
Plus there's a lot more here:
https://movablefeastmachine.org/
Now he's working on a hardware implementation of indefinitely scalable robust first computing:
https://www.youtube.com/channel/UC1M91QuLZfCzHjBMEKvIc-A
David Ackley, who developed the two-dimensional CA-like "Moveable Feast Machine" architecture for "Robust First Computing", touched on moving from 2D to 3D in his retirement talk:
https://youtu.be/YtzKgTxtVH8?t=3780
"Well 3D is the number one question. And my answer is, depending on what mood I'm in, we need to crawl before we fly."
"Or I say, I need to actually preserve one dimension to build the thing and fix it. Imagine if you had a three-dimensional computer, how you can actually fix something in the middle of it? It's going to be a bit of a challenge."
"So fundamentally, I'm just keeping the third dimension in my back pocket, to do other engineering. I think it would be relatively easy to imaging taking a 2D model like this, and having a finite number of layers of it, sort of a 2.1D model, where there would be a little local communication up and down, and then it was indefinitely scalable in two dimensions."
"And I think that might in fact be quite powerful. Beyond that you think about things like what about wrap-around torus connectivity rooowaaah, non-euclidian dwooraaah, aaah uuh, they say you can do that if you want, but you have to respect indefinite scalability. Our world is 3D, and you can make little tricks to make toruses embedded in a thing, but it has other consequences."
Here's more stuff about the Moveable Feast Machine:
https://news.ycombinator.com/item?id=15560845
https://news.ycombinator.com/item?id=14236973
The most amazing mind blowing demo is Robust-first Computing: Distributed City Generation:
https://www.youtube.com/watch?v=XkSXERxucPc
And a paper about how that works:
https://www.cs.unm.edu/~ackley/papers/paper_tsmall1_11_24.pd....
Plus there's a lot more here:
https://movablefeastmachine.org/
Now he's working on a hardware implementation of indefinitely scalable robust first computing:
https://www.youtube.com/channel/UC1M91QuLZfCzHjBMEKvIc-A
I think there is a severe lack of imagination going on here. A pencil is a 3D interface.
This is as good a time as any to post this eight-year old gem from Bret Victor: http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...
This is as good a time as any to post this eight-year old gem from Bret Victor: http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...
A keyboard is also a 3D interface with one dimension constrained. I have to be able to move my hand past the keyboard without accidentally inputting data. You can see the problems of this kind when you have a large touch screen and people try to gesture at it while talking and end up accidentally changing something.
Look at machine rooms in submarines or control rooms for power plants or tv studios. Different tasks are in different locations sure, but the inputs are constrained to an area near the arc defined by the fingers at a comfortable distance from the shoulder. For really good reasons.
Look at machine rooms in submarines or control rooms for power plants or tv studios. Different tasks are in different locations sure, but the inputs are constrained to an area near the arc defined by the fingers at a comfortable distance from the shoulder. For really good reasons.
> A pencil is a 3D interface.
Is it? I’d say the VR analogue of a pencil is a pointing device, which makes the user interface the VR analogue of paper. And paper is fundamentally a 2D interface. There are some 3D actions like turning a page, but those are secondary and rare. You can move and rotate the paper as a whole in 3D, and that’s useful, but the same functionality can easily be added to VR 2D interfaces.
On the other hand, there are other types of art that are fundamentally 3D, such as sculpture and pottery – but both of those rely on feeling a 3D object with your hands, which partially bypasses the 2D limitation of vision, but isn’t yet possible to emulate in VR.
Then again, there’s also the common VR toy of a “pencil” that doodles in thin air, which is certainly interesting... though I’m not sure how well it generalizes to more abstract user interfaces. If you’re using such a system, you have to constantly rotate the object you’re drawing, and/or move your body around, in order to properly perceive the object in 3D. This is kind of a pain. If your primary goal is to create a 3D object, it’s an unavoidable pain and the benefit is well worth it; but if what you’re interacting with is just an abstract interface meant to manipulate something non-spatial, it’s probably better to avoid.
Is it? I’d say the VR analogue of a pencil is a pointing device, which makes the user interface the VR analogue of paper. And paper is fundamentally a 2D interface. There are some 3D actions like turning a page, but those are secondary and rare. You can move and rotate the paper as a whole in 3D, and that’s useful, but the same functionality can easily be added to VR 2D interfaces.
On the other hand, there are other types of art that are fundamentally 3D, such as sculpture and pottery – but both of those rely on feeling a 3D object with your hands, which partially bypasses the 2D limitation of vision, but isn’t yet possible to emulate in VR.
Then again, there’s also the common VR toy of a “pencil” that doodles in thin air, which is certainly interesting... though I’m not sure how well it generalizes to more abstract user interfaces. If you’re using such a system, you have to constantly rotate the object you’re drawing, and/or move your body around, in order to properly perceive the object in 3D. This is kind of a pain. If your primary goal is to create a 3D object, it’s an unavoidable pain and the benefit is well worth it; but if what you’re interacting with is just an abstract interface meant to manipulate something non-spatial, it’s probably better to avoid.
> both of those rely on feeling a 3D object with your hands, which partially bypasses the 2D limitation of vision, but isn’t yet possible to emulate in VR.
That's the entire point of the comment you're responding to and the reason for Dynamicland.
That's the entire point of the comment you're responding to and the reason for Dynamicland.
If so, it was off topic. John Carmack's post was about VR interfaces, and I interpreted the parent comment in that context. In any case, I would contest that bypassing the limitations of vision, specifically, represents a significant part of the reason for Dynamicland.
If you "would contest" that, it's because you still haven't read the essay that was linked in the putatively off-topic comment you were "responding" to, after eight years. And, yeah, you could try to redefine "VR" and even "3D interfaces" (the actual topic) as being strictly limited to "binocular video responding to a head tracker", but even "VR" researchers have been researching haptic and kinesthetic feedback from a zillion angles for a lot more than eight years, so I don't think it's off-topic at all.
> I’d say the VR analogue of a pencil is a pointing device, which makes the user interface the VR analogue of paper
In this day and age, when accelerometers can be embedded in small objects, why don't we stop using analogues and just design a real world smart pencil that be used to control the VR floating pencil?
In this day and age, when accelerometers can be embedded in small objects, why don't we stop using analogues and just design a real world smart pencil that be used to control the VR floating pencil?
Because it's not a natural interface?
Given there are is no touch feedback when you feel surfaces in VR and that just holding your arms up in the arm for long periods of times is tiring, I honestly don't see why it's any better than pointing on a 2D surface.
People wanted to do Minority Report-style UIs when they saw them, but we generally don't interact with computers in those ways for the same reason. Keyboard and mouse (or trackpad) is going to be hard to improve upon.
Given there are is no touch feedback when you feel surfaces in VR and that just holding your arms up in the arm for long periods of times is tiring, I honestly don't see why it's any better than pointing on a 2D surface.
People wanted to do Minority Report-style UIs when they saw them, but we generally don't interact with computers in those ways for the same reason. Keyboard and mouse (or trackpad) is going to be hard to improve upon.
We have 3d output capabilities, we maneuver our hands in 3 dimensions. As jerf explains in a comment above better than I can we only have 2d input capabilities.
I interpret this post as primarily saying that the UI's display (as in what is fed to our input) should only be 2d, not necessarily the other way around.
https://news.ycombinator.com/item?id=19963247
Edit: Linked wrong comment, fixed.
I interpret this post as primarily saying that the UI's display (as in what is fed to our input) should only be 2d, not necessarily the other way around.
https://news.ycombinator.com/item?id=19963247
Edit: Linked wrong comment, fixed.
The idea that we only have 2D input doesn't make sense to me. If that were the case, how would we drive a car or ride a bike? You don't need to jump between focusing on things that are far away and up close, as Carmack says. It's totally natural.
I would argue that humans, like most mammals, are actually most at ease in an immersive 3D medium.
And in theory, the only thing stopping us from implementing something like Bret Victor's Dynamicland (https://dynamicland.org) in VR is the lack of good 3D input methods, like say a pair of sensor-ridden smart gloves.
John Carmack's argument reminds me of the early criticisms of the point-and-click interface (2D), and how at its inception it was much less efficient than the well developed command line interface (1D).
Plus, the most designers are trained in 2D interfaces so they're probably applying the wrong assumptions for 3D.
I would argue that humans, like most mammals, are actually most at ease in an immersive 3D medium.
And in theory, the only thing stopping us from implementing something like Bret Victor's Dynamicland (https://dynamicland.org) in VR is the lack of good 3D input methods, like say a pair of sensor-ridden smart gloves.
John Carmack's argument reminds me of the early criticisms of the point-and-click interface (2D), and how at its inception it was much less efficient than the well developed command line interface (1D).
Plus, the most designers are trained in 2D interfaces so they're probably applying the wrong assumptions for 3D.
2D + depth. We can tell how far away the car in front of us is, but it occludes our view of cars in front of it. That’s good enough to drive, but suboptimal. If you were designing a user interface meant to show someone the positions of cars on roads – i.e. a map – you would use a bird’s-eye view, since roads are mostly 2D from that perspective.
You can drive a car without depth perception. You don't need to see the depth of cars/obstacles/the road if it can be deduced from geometry.
Point taken, but then why go through all the trouble to evolve two eyes? Just in case one gets poked or something?
I bet one getting poked has more to do with it than you think. We have two lungs too after all and they're a lot less exposed than your eyes. I use that redundancy with a disturbing amount of regularity when biking too... with something getting in one eye and temporarily disabling it (I really need to look into some form of eye protection from road dust).
But thinking of depth in vision like color also makes a lot of sense, it's extra information but not an extra "physical" dimension. You can't see inside a sphere but you can a circle.
But thinking of depth in vision like color also makes a lot of sense, it's extra information but not an extra "physical" dimension. You can't see inside a sphere but you can a circle.
I can think of even more ways we benefit from having two eyes, like not having to whip your head back and forth if you want to scan the landscape.
But if they were mainly for redundancy the eyes might be closer together, like the nostrils (and unlike ears, which help with spatial awareness).
I think the fact that modern life requires less and less depth perception (when’s the last time you had to judge how far to throw a spear, etc) might be leading us astray here.
But if they were mainly for redundancy the eyes might be closer together, like the nostrils (and unlike ears, which help with spatial awareness).
I think the fact that modern life requires less and less depth perception (when’s the last time you had to judge how far to throw a spear, etc) might be leading us astray here.
Great read. Really highlights all the problems I have with touch interfaces. Unfortunately I don't see it changing much in the near-future mainly because it's Good Enough for most people and most operations. Wish someone would put money into researching tactile screen interfaces that physically deform and respond to touch.
[deleted]
Yes... but no...
Resemblance to reality as UI design smell... Just because we have head tracking and stereo display, doesn't mean a UI should at all resemble reality. Years ago, when cell phones were new, and UI design was focused on onboarding novice users in an untrained culture, there was the idea of skeuomorphic ui design - the app should resemble the familiar physical thing. The calendar app should have lined pages, that flip around a spiral binding, surrounded by a textured leather margin. We don't do that anymore. It would be silly. As is 3D UI's resembling "spear chucking" reality. So yes, tossing xterms in the air, and 3D UI's that resemble VR games, are both common and bad ideas. Quietly assuming VR-gaming-like things cover the entire 3D UI design space, and drawing conclusions built on that assumption, is another common and bad idea.
> Splitting information across multiple depths is harmful
Frequently jumping between dissimilar depths is harmful. Less frequent, sliding, and similar depths, can be wonderful, allowing much denser and easily accessible presentation of information. I so wish my laptop had a native 3D display... DIY anaglyph and shutter glasses are a pain.
A general takeaway is... Much current commentary about "VR", is coming from a community focused on a particular niche, current VR gaming. One with particular and severe, constraints and priorities... that very don't characterize the entirety of a much larger design space. And that community, and commentary, has been really bad at maintaining awareness of that. So it's on you.
Resemblance to reality as UI design smell... Just because we have head tracking and stereo display, doesn't mean a UI should at all resemble reality. Years ago, when cell phones were new, and UI design was focused on onboarding novice users in an untrained culture, there was the idea of skeuomorphic ui design - the app should resemble the familiar physical thing. The calendar app should have lined pages, that flip around a spiral binding, surrounded by a textured leather margin. We don't do that anymore. It would be silly. As is 3D UI's resembling "spear chucking" reality. So yes, tossing xterms in the air, and 3D UI's that resemble VR games, are both common and bad ideas. Quietly assuming VR-gaming-like things cover the entire 3D UI design space, and drawing conclusions built on that assumption, is another common and bad idea.
> Splitting information across multiple depths is harmful
Frequently jumping between dissimilar depths is harmful. Less frequent, sliding, and similar depths, can be wonderful, allowing much denser and easily accessible presentation of information. I so wish my laptop had a native 3D display... DIY anaglyph and shutter glasses are a pain.
A general takeaway is... Much current commentary about "VR", is coming from a community focused on a particular niche, current VR gaming. One with particular and severe, constraints and priorities... that very don't characterize the entirety of a much larger design space. And that community, and commentary, has been really bad at maintaining awareness of that. So it's on you.
Do you have links to example 3D UIs that you're talking about?
Sorry, I've not seen such.
And my own stuff was so very limited and crufty, before I rather burned out on the yak shaving of it all. Though two requests for demos just this evening... sigh.
Perhaps picture the cylindrical "big screen" someone here mentioned. Run 3D programs like CAD in some of the windows. Control the window manger with keystrokes. Use a trackpad, a mouse, a 3D mouse. These inputs aren't tightly coupled to the rendered 3D space. Add a 6DOF controller. There's still no need to introduce a tight coupling - it can sit beside your mouse, or on your knee, and twitch. Discard the gaming objective of immersion, and show the outside world. Use a non-standard VR rendering stack, or AR, so that's performant. Open up the HMD horse blinders. Add some "comfort mode" tricks, and now the outside world is being used for balance, and the rendered world is freed to be bizarre. When your head moves, move the cylinder twice as fast, to reduce the need for head motion. Turn it with keys or mouse and no head motion. It's not a real cylinder - there's no reason when you turn it and turn it back, the same thing has to be there - it can be whatever you want, whatever is helpful. The motivation for a surface is coplanarity for optical scanning, and living in some hardware/wetware sweet spot. But one is a task-specific constraint, and both are local to the region currently shown, not global constraints. So punt the cylinder, and use an arbitrary aphysical topology. So now, while do you have a 3D visual display, and 3D HIDs, the contents and inputs are no more constrained to some imagined reality than in any usual window manager an apps. Choosing how much resemblance to reality to exhibit, as with scrolling "physics", is simply a UI design choice. ... Ah well - it's late - FWIW.
And my own stuff was so very limited and crufty, before I rather burned out on the yak shaving of it all. Though two requests for demos just this evening... sigh.
Perhaps picture the cylindrical "big screen" someone here mentioned. Run 3D programs like CAD in some of the windows. Control the window manger with keystrokes. Use a trackpad, a mouse, a 3D mouse. These inputs aren't tightly coupled to the rendered 3D space. Add a 6DOF controller. There's still no need to introduce a tight coupling - it can sit beside your mouse, or on your knee, and twitch. Discard the gaming objective of immersion, and show the outside world. Use a non-standard VR rendering stack, or AR, so that's performant. Open up the HMD horse blinders. Add some "comfort mode" tricks, and now the outside world is being used for balance, and the rendered world is freed to be bizarre. When your head moves, move the cylinder twice as fast, to reduce the need for head motion. Turn it with keys or mouse and no head motion. It's not a real cylinder - there's no reason when you turn it and turn it back, the same thing has to be there - it can be whatever you want, whatever is helpful. The motivation for a surface is coplanarity for optical scanning, and living in some hardware/wetware sweet spot. But one is a task-specific constraint, and both are local to the region currently shown, not global constraints. So punt the cylinder, and use an arbitrary aphysical topology. So now, while do you have a 3D visual display, and 3D HIDs, the contents and inputs are no more constrained to some imagined reality than in any usual window manager an apps. Choosing how much resemblance to reality to exhibit, as with scrolling "physics", is simply a UI design choice. ... Ah well - it's late - FWIW.
Not OP but the team at Leap Motion [1] has some very interesting demos. 3D UI's re-imagined from the ground up. Weirdly, because now everything is 3D, the best you can do is try to mimic actually 3D things perhaps, something that didn't work in 2D at all?
[1] https://www.youtube.com/watch?v=7m6J8W6Ib4w
[2] https://www.youtube.com/watch?v=6dB1IRg3Qls
[1] https://www.youtube.com/watch?v=7m6J8W6Ib4w
[2] https://www.youtube.com/watch?v=6dB1IRg3Qls
They're fun demos. And perhaps from the perspective of say mobile, they're kind of reasonable. Rotating your wrist? Waving your arm to press a button? Waving your arm to move a window? Not all that odd when holding a phone.
But from the perspective of laptops/desktops? What comes to my mind is: RSI; slowwwww, but maybe it's the touchpad version of a standing desk?; people use tiling window managers in part because waving a finger to place windows in 2D is already too burdensome.
I've just now failed again to find nice video, but if you've ever seen it, picture a professional artist tooling along on a Cintiq at insane speed. Stylus, and two handed multitouch flying. Zipping through enormous menus. Twitch gesturing through radial menus. Stylus input with pressure and tilt and rotation. Now imagine that interactive area was several centimeters high. And imagine transparent layers of content above the screen (you might sort of get a feel, by moving a semi-transparent window in front of another, and switch between reading one and the other, back and forth). Imagine a 3D visual environment extending below and above the screen. And around it, throughout the room. Now imagine your keyboard does multitouch, and instead of art, that's your code.
But from the perspective of laptops/desktops? What comes to my mind is: RSI; slowwwww, but maybe it's the touchpad version of a standing desk?; people use tiling window managers in part because waving a finger to place windows in 2D is already too burdensome.
I've just now failed again to find nice video, but if you've ever seen it, picture a professional artist tooling along on a Cintiq at insane speed. Stylus, and two handed multitouch flying. Zipping through enormous menus. Twitch gesturing through radial menus. Stylus input with pressure and tilt and rotation. Now imagine that interactive area was several centimeters high. And imagine transparent layers of content above the screen (you might sort of get a feel, by moving a semi-transparent window in front of another, and switch between reading one and the other, back and forth). Imagine a 3D visual environment extending below and above the screen. And around it, throughout the room. Now imagine your keyboard does multitouch, and instead of art, that's your code.
John Carmack:
> Last year, I argued that cylindrical panels were surprisingly effective, and we should embrace them for VR Shell UI. There was a lot of push back about giving up the flexibility of putting UI wherever you want in 3D, and curtailing the imagined future world of even more 3D interfaces, but the argument for objectively higher quality with the native TimeWarp layer projection kept it from being a completely abstract design question.
Perhaps something similar to this in 2D interfaces: I miss the original iOS’ embossed buttons with highlights and shadows that made their clickable-ness clear, and find the text links we have now inferior. Older people I know who have used iOS from the beginning also lament this change.
> Last year, I argued that cylindrical panels were surprisingly effective, and we should embrace them for VR Shell UI. There was a lot of push back about giving up the flexibility of putting UI wherever you want in 3D, and curtailing the imagined future world of even more 3D interfaces, but the argument for objectively higher quality with the native TimeWarp layer projection kept it from being a completely abstract design question.
Perhaps something similar to this in 2D interfaces: I miss the original iOS’ embossed buttons with highlights and shadows that made their clickable-ness clear, and find the text links we have now inferior. Older people I know who have used iOS from the beginning also lament this change.
Yes, many modern UIs are anti-UX dark-patterns of past.
I worked in a media tech lab that experimented with how to tell stories in VR. The truth we encountered was that there was very little that VR did to enhance storytelling. There's a lot of hyperbolic arguments that sound good, but don't pan out in practice.
Reading Carmack's argument does make me wonder how much is objectively true and how much is because we're trying to bootstrap a generation of 2D interface users into 3D. Will this be one of those things that a generation of kids raised in VR will roll their eyes at? I don't know.
Reading Carmack's argument does make me wonder how much is objectively true and how much is because we're trying to bootstrap a generation of 2D interface users into 3D. Will this be one of those things that a generation of kids raised in VR will roll their eyes at? I don't know.
I generally agree.
I like keyboard interfaces for non-spatial input over a mouse because fingers require less effort to move than arms and wrists. They can also move faster. These problems are made worse in 3D.
I see 3D being useful in 2 cases:
1) Adding depth allows more keeping of context. This is similar to zooming in and out by pinching to keep the context. But with 3D you lose less peripheral view and occlude less of the surround content. Our brains are better wired for spatial processing in 3D.
2) Fitt's Law predicts that the time required to rapidly move to a target area is a function of the ratio between the distance to the target and the width of the target. By adding another dimensions we can get objects closer and reduce the distance.
I like keyboard interfaces for non-spatial input over a mouse because fingers require less effort to move than arms and wrists. They can also move faster. These problems are made worse in 3D.
I see 3D being useful in 2 cases:
1) Adding depth allows more keeping of context. This is similar to zooming in and out by pinching to keep the context. But with 3D you lose less peripheral view and occlude less of the surround content. Our brains are better wired for spatial processing in 3D.
2) Fitt's Law predicts that the time required to rapidly move to a target area is a function of the ratio between the distance to the target and the width of the target. By adding another dimensions we can get objects closer and reduce the distance.
Cylindrical panels, TimeWarp, Medium, Quill
What is he talking about? Do Facebook posts not allow links?
What is he talking about? Do Facebook posts not allow links?
Cylindrical panels - I believe he's talking about positioning 2D surfaces along the inside of a imaginary cylinder centered around the user. Like this: https://youtu.be/SvP_RI_S-bw?t=7
TimeWarp - A technology for reducing percieved latency in VR. https://uploadvr.com/reprojection-explained/
Medium - A VR sculpting app. https://www.oculus.com/medium/
Quill - A VR drawing app. https://www.oculus.com/experiences/rift/1118609381580656/
TimeWarp - A technology for reducing percieved latency in VR. https://uploadvr.com/reprojection-explained/
Medium - A VR sculpting app. https://www.oculus.com/medium/
Quill - A VR drawing app. https://www.oculus.com/experiences/rift/1118609381580656/
I can't watch the video now, but does it explain why the inside of a cylinder rather than the inside of a sphere? This was my question after reading the OP; if you truly want the UI to be equidistant from a single point (the user's head) in two dimensions, it seems like a sphere is the obvious choice there.
My guess here would not be to have the entire interface equidistant from the user, but, rather, to have the equivalent of a wrap around monitor.
We're already used to having flat screens to interact with and simply curving the display doesn't, in my experience with a curved monitor, have any detrimental effect on your ability to view the display.
Beyond that, in a mathematical sense, a square and a cylinder are very topologically similar. Of note is that the relative size and shape of things on the surface are well preserved[1] as you don't need to significantly deform a square to turn it into a cylinder.
This is not the case for a sphere.
As a practical demonstration, take a piece of paper and draw some boxes on it before wrapping it around a reasonably sized ball. You'll see that the boxes (particularly the larger boxes) become deformed in a way that likely will make them seem to "bulge out". As far as I'm aware, the effect happens to any regular polygon wrapped around a sphere. The same thing will happen to any interface that attempts to project onto a sphere.
So, given that:
a) we're already used to flat interfaces
b) curving that interface into a semi-cylinder doesn't[2] have any detrimental effects
c) that traditional square interfaces don't map directly/cleanly onto a sphere
It's easy to conclude that a cylindrical interface is probably the simplest to make and easiest to understand interface. It requires the smallest amount of work to make and the least learning on the part of the user.
1: Obviously, things on the cylinder will be curved, but the 2 dimensional shape is still consistent.
2: for me, at least
We're already used to having flat screens to interact with and simply curving the display doesn't, in my experience with a curved monitor, have any detrimental effect on your ability to view the display.
Beyond that, in a mathematical sense, a square and a cylinder are very topologically similar. Of note is that the relative size and shape of things on the surface are well preserved[1] as you don't need to significantly deform a square to turn it into a cylinder.
This is not the case for a sphere.
As a practical demonstration, take a piece of paper and draw some boxes on it before wrapping it around a reasonably sized ball. You'll see that the boxes (particularly the larger boxes) become deformed in a way that likely will make them seem to "bulge out". As far as I'm aware, the effect happens to any regular polygon wrapped around a sphere. The same thing will happen to any interface that attempts to project onto a sphere.
So, given that:
a) we're already used to flat interfaces
b) curving that interface into a semi-cylinder doesn't[2] have any detrimental effects
c) that traditional square interfaces don't map directly/cleanly onto a sphere
It's easy to conclude that a cylindrical interface is probably the simplest to make and easiest to understand interface. It requires the smallest amount of work to make and the least learning on the part of the user.
1: Obviously, things on the cylinder will be curved, but the 2 dimensional shape is still consistent.
2: for me, at least
You can scroll up and down in cylinder of infinite height content, like you can scroll up and down an infinite height web page with a smartphone.
Mathematically speaking, the inside of a cylinder has an infinite area, while the inside of a sphere has a fixed area. So there's room for more stuff!
Mathematically speaking, the inside of a cylinder has an infinite area, while the inside of a sphere has a fixed area. So there's room for more stuff!
I imagine that one reason for the cylinder instead of the sphere is that it is much easier/more comfortable for a human to shift the angle of the head in the horizontal plane than in the vertical plane.
Well a cylinder better fits rectangular content, like images.
No, the video doesn't contain any additional explanation; it's just an example of a real-world implementation of the idea in Oculus's VR interface.
I couldn't find any additional information on why they didn't choose a sphere, though perhaps it has something to do with the additional distortion that would be created by bending the UI in two directions at the same time.
I couldn't find any additional information on why they didn't choose a sphere, though perhaps it has something to do with the additional distortion that would be created by bending the UI in two directions at the same time.
The sphere has all kinds of distortions. You can map a flat paper (image, website, ...) on a cylinder, but not on a sphere.
I think most of us who get into data visualization go through a "3D all the things!" phase. I certainly did. But eventually I had to accept that the tradeoffs just aren't worth it. Occlusion is a serious problem. Our brains simply process 2D much more efficiently.
Obviously volumetric data is an exception.
Obviously volumetric data is an exception.
Well, yes. Ever since the old SGI 3D file browser [1], a true 3D interface approach has been terrible. Tilt Brush is probably the best success so far.[2] Even that, though, isn't very good for artistic work. It's just finger-painting in 3D.
Without force feedback, human positioning in 3D space is terrible. Maybe if we get usable VR gloves with force feedback, so you can grab a knob or lever and feel the movement, stops, and detents, it could be workable.
[1] https://en.wikipedia.org/wiki/Fsn_(file_manager)
[2] https://www.youtube.com/watch?v=TckqNdrdbgk
Without force feedback, human positioning in 3D space is terrible. Maybe if we get usable VR gloves with force feedback, so you can grab a knob or lever and feel the movement, stops, and detents, it could be workable.
[1] https://en.wikipedia.org/wiki/Fsn_(file_manager)
[2] https://www.youtube.com/watch?v=TckqNdrdbgk
Check out this Smarter Every Day introduction to the HaptX force feedback glove from last year.[1] The guy thought VR was stupid until he used the glove, then was instantly immersed. They’ve packed a lot of sensory feedback into it.
Once force feedback hits, translating 2D onscreen control surfaces to 3D models will be both trivial and worthwhile, at least in the niche realm of virtual audio devices. Touchscreens have already made them much more playable, and things like TouchOSC allow for some customization. I’d love to have something like the Reason rack[2] in a space where I could use all 10 fingers instead of 1 mouse pointer to interact with it.
[1] https://youtu.be/OK2y4Z5IkZ0
[2] https://images.app.goo.gl/37FMUZ6kfgTb5rPf6
Once force feedback hits, translating 2D onscreen control surfaces to 3D models will be both trivial and worthwhile, at least in the niche realm of virtual audio devices. Touchscreens have already made them much more playable, and things like TouchOSC allow for some customization. I’d love to have something like the Reason rack[2] in a space where I could use all 10 fingers instead of 1 mouse pointer to interact with it.
[1] https://youtu.be/OK2y4Z5IkZ0
[2] https://images.app.goo.gl/37FMUZ6kfgTb5rPf6
> Splitting information across multiple depths is harmful, because your eyes need to re-verge (and, we wish, re-focus).
> It may be worse in VR, because you have to fight the lack of actual focus change. With varifocal we would be back to just the badness of the real world, but no better. This is fundamental to the way humans work.
This point on the eye focus changes caused while viewing 3D interfaces (as opposed to 2D interfaces) is a very well-founded one.
This point on the eye focus changes caused while viewing 3D interfaces (as opposed to 2D interfaces) is a very well-founded one.
the key is to use a 3d interface when the data is actually 3d, and only then.
3d: Level designing / modeling, physics, anything with spatial relationships
2d: Options menu, scripting and logic, numbers and info
since we read in 2d it's always going to be easier to parse
since we read in 2d it's always going to be easier to parse
He's right that changing your depth focus is more taxing, but I wonder what's healthiest for the eyes? Maybe it would be good to for the eyes to change focus depth while wearing VR headsets? If I stare at my monitor too long without breaks my eyes definitely get lethargic while trying to focus into the distance. (General tip: every 20 minutes stare at something at least 20 feet away for 20 seconds.)
Curved monitors are basically meeting the same need - to maintain the same focal depth as one looks side to side, shifting between windows, applications, etc.
I did find a few papers with a quick search that covers the topic for monitors. I would assume there is some carryover of any kind of finding about curved monitor ergonomics to the cylindrical view idea in VR.
See https://pdfs.semanticscholar.org/d1c0/3f54a46b5fefc27b7a4207... and https://www.tandfonline.com/doi/pdf/10.1080/15980316.2015.11... for example.
I did find a few papers with a quick search that covers the topic for monitors. I would assume there is some carryover of any kind of finding about curved monitor ergonomics to the cylindrical view idea in VR.
See https://pdfs.semanticscholar.org/d1c0/3f54a46b5fefc27b7a4207... and https://www.tandfonline.com/doi/pdf/10.1080/15980316.2015.11... for example.
It's really interesting to see Carmack post this opinion (especially as it is two years old). I trust his insight and way of doing things, so it's fascinating to see even a glimpse of how he operates in a larger company where clearly their objectives and opinions won't always match his own.
I wonder in general if he's still bullish on Oculus (perhaps he is with VR in general though).
I wonder in general if he's still bullish on Oculus (perhaps he is with VR in general though).
I having been using the cylindrical layout shown here:
https://arcan-fe.com/2018/03/29/safespaces-an-open-source-vr... desktop/
for a few weeks now - far from full time as the HMD display, lenses (CV1 here) and positioning/rotation tracking are not good enough with the open source driver but it feels really close. Except for some tweaking and extra "spacing" between the focus window and the side windows, it works exactly as I want it to. I added some customisation so the 'layers' do not exist at different depths but on different arcs on the same layer. Switching "workspaces" simply rotates these layers to the 12 o clock position. Biggest problem right now is not having front cameras to see what my hands are doing.
https://arcan-fe.com/2018/03/29/safespaces-an-open-source-vr... desktop/
for a few weeks now - far from full time as the HMD display, lenses (CV1 here) and positioning/rotation tracking are not good enough with the open source driver but it feels really close. Except for some tweaking and extra "spacing" between the focus window and the side windows, it works exactly as I want it to. I added some customisation so the 'layers' do not exist at different depths but on different arcs on the same layer. Switching "workspaces" simply rotates these layers to the 12 o clock position. Biggest problem right now is not having front cameras to see what my hands are doing.
One interesting fenomene is that focused 2d images appear to be more high res. It would be cool if a 3d UI vould somehow know in real time what we focus on and put that plane in focus. Somehow we can see if another human or animal is focusing in on us, so it might be possible for a computer to also see what we focus on.
media molecule's Dreams has the most impressive user interface i've ever used. it uses 2d panels laid out in 3d space but the panels can be anchored and resized in screen space, object space or world space and it's extremely intuitive. i highly recommend people interested in ux/ui check it out
Looks very inefficient.
Pressing a key on a keyboard works better than using spatial input from a pointer device for those cases. To click a button/icon there is a feedback loop that requires vision processing, correction, anticipation, and hand eye coordination. You have to iterate many times in this loop to move to the correct location. Often the refresh latency makes this problem even harder.
This problem is better understood when you have a virtual keyboard and only a single pointer.
Ctrl/Cmd-C is easier and faster than moving the mouse to Edit -> Copy.
Creative professionals rely on physical keyboards and buttons on the mouse for actions and only use spatial input for things that are spatially relevant (pan, placement, zoom, rotation, selecting surfaces, edges, vertices, etc).
Pressing a key on a keyboard works better than using spatial input from a pointer device for those cases. To click a button/icon there is a feedback loop that requires vision processing, correction, anticipation, and hand eye coordination. You have to iterate many times in this loop to move to the correct location. Often the refresh latency makes this problem even harder.
This problem is better understood when you have a virtual keyboard and only a single pointer.
Ctrl/Cmd-C is easier and faster than moving the mouse to Edit -> Copy.
Creative professionals rely on physical keyboards and buttons on the mouse for actions and only use spatial input for things that are spatially relevant (pan, placement, zoom, rotation, selecting surfaces, edges, vertices, etc).
Creative professionals rely on physical keyboards and buttons on the mouse for actions and only use spatial input for things that are spatially relevant (pan, placement, zoom, rotation, selecting surfaces, edges, vertices, etc).
That used to be true. Autodesk put a lot of effort into interfaces for engineering in 3D. In Inventor, you only need the keyboard to enter numbers or names. They managed to do it all with the mouse. Try Fusion 360 to see this; there's a free demo.
That used to be true. Autodesk put a lot of effort into interfaces for engineering in 3D. In Inventor, you only need the keyboard to enter numbers or names. They managed to do it all with the mouse. Try Fusion 360 to see this; there's a free demo.
You mean the right-click radial context menus? Yes, those are really nice. They require much less of a feedback loop and often don't require any visual processing at all. They are also more gesture based not buttons. The affordance is very generous compared to buttons and icons.
I use a 3DConnexion Space Navigator (6 DOF) in my left hand and mouse in my right for selection when using Fusion 360 and often use the gestures on the mouse.
I guess that brings up an exception. The context switching cost. Moving from pointer to keyboard is very slow so gestures really help out in that regard. If my hand is already on the keyboard then I have less reason to want to use the gestures.
I use a 3DConnexion Space Navigator (6 DOF) in my left hand and mouse in my right for selection when using Fusion 360 and often use the gestures on the mouse.
I guess that brings up an exception. The context switching cost. Moving from pointer to keyboard is very slow so gestures really help out in that regard. If my hand is already on the keyboard then I have less reason to want to use the gestures.
That, plus the ability to rotate, pan and zoom while you're in the middle of a selection. That's a huge win. In 3D work, you often need to select 2 or more things, and those things may be small and need precise selection. Precision multiple selection is hard.
Before this was worked out, most 3D programs offered four panes, with three axial projections, usually wireframe, plus a solid perspective view. Just so you could select. Now we only need one big 3D pane.
Before this was worked out, most 3D programs offered four panes, with three axial projections, usually wireframe, plus a solid perspective view. Just so you could select. Now we only need one big 3D pane.
you really dont use context menus for everything. a lot of functionality is based on modal key chords. so, different sequences/chords do different things based on the selected mode. watch this video and tell me this is inefficient: https://youtu.be/31A0s9HDRHU?t=483
It probably needs to be pointed out that this is a post from John Carmack. I almost didn't click on it due to the facebook.com domain but spotted his name in the comments here. Obviously Carmack is still highly regarded and respected in the industry and his opinions quite rightly hold a lot of weight, despite his current choice of employer / platform.
Here's another nice article from June 2018 touching on many of the same points:
https://hackernoon.com/predictions-from-the-first-ar-os-9440...
>The way we organise our physical world is probably more relevant to AR than how we organise our 2D interfaces. ...
https://hackernoon.com/predictions-from-the-first-ar-os-9440...
>The way we organise our physical world is probably more relevant to AR than how we organise our 2D interfaces. ...
I first misunderstood this as interfaces in 3D games, and it doesn’t really change the supposition.
Interacting with most AAA title UI is an exercise in extreme frustration. Days Gone’s menu is like someone developed a fetish for prezto.js.
It’s pretty, but also pretty unintuitive.
Also beside the point a bit.
Interacting with most AAA title UI is an exercise in extreme frustration. Days Gone’s menu is like someone developed a fetish for prezto.js.
It’s pretty, but also pretty unintuitive.
Also beside the point a bit.
I feel like this needs examples. What exactly is he recommending as a gold standard?
2d interface on the inside of a cylinder that surrounds the user, it seems. Perhaps not all the way around.
That makes sense -- switching depths is more taxing (and gets weirder in VR, since you're not actually changing your eyes' point of focus), but a 2D cylinder gives you as much horizontal space as you could possibly get.
[deleted]
I was angered by the title (a bit clickbaity, as context is king for a UI), but upon reading agree 100% with the title.
Of course. We can't see 3 dimensions; we can only see what can be projected onto a portion of a spheroid.
The point is abstraction, more sensory dimensions is very unlikely to provide there.
Imagine if we interacted with everything we encounter every day on a single 2D plane.
Facebook?! John Carmack doesn't have a blog? :(
John currently works for Facebook as CTO of Oculus.
thats too bad as their interface is absolute garbage.
He works for Facebook (via Occulus).
That said, I do wish he'd put out the blog posts as .plan files as well.
That said, I do wish he'd put out the blog posts as .plan files as well.
He works for Facebook.
He probably doesn't care if non-Facebook people don't read his posts.
[deleted]
Maps are one very good example of this. A map is partially so powerful because it removes a ton of information you probably don't care about. Occasionally would you like to know a house is up a steep hill before you drive there? Sure. But most of the time who cares. And if you really need elevation data, there's a special map for that. A map which actually shows you information about grade in much clearer way than even a 3D model would (being as humans are actually really bad at guessing a grade by looking at a real slope).