Hi - I'm Simon, and I've been leading the development on the low-level runtime bits for Zapbox.
We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon").
Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience.
The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement.
The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do.
What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :)
[1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.
OP here - I'm hoping by posting a forum thread explicitly stating their current behaviour is broken and providing a link to the spec that they are violating will allow Amazon to accept they have an issue and hopefully fix it.
There are so many threads on CORS around StackOverflow and the AWS forums and users tend to assume they've got something wrong in their configuration, or blame the browsers for returning a cached response that they are completely entitled to do. "Solutions" are always given to ensure all requests are cross origin (such as by `<img crossorigin ...`) or cache-busting with a query parameter.
S3's CORS implementation came out in 2012 and has never managed to get this right as far as I can tell. They did add a `Vary: Origin` header to their CORS responses in April 2013 - https://forums.aws.amazon.com/thread.jspa?threadID=103402&st... - but still now over 8 years later their non-CORS responses don't have this header.
No worries, I imagine many teams have been through the same process over the past couple of weeks!
We've definitely had the same experience of Safari introducing way more issues with new releases than Chrome, and the fact it's part of the iOS image seems to make quick patching just never really an option.
I just wish they'd engage a bit more directly on the bug tracker - find it hard to believe that twitter is a more effective means to get attention on bugs.
Chrome 91 on Android also managed to break WebAssembly memory growth which affects our sites too, so it's been a rough couple of weeks working around browser bugs for us.
https://bugs.chromium.org/p/v8/issues/detail?id=11863
I find the Chromium team much more willing to engage directly and constructively on their bug trackers at least.
I wrote up the bug report for this a couple of weeks ago when iOS 14.6 first rolled out. iOS 14.6 at least did finally fix using HLS streams in gl.texUpload2D() on iPhone 12 series (which was broken since launch) but this new bug did mean our delight at that fact was somewhat short-lived!
Well done on your viral tweet that got some personal attention from Apple. Usually when I spend days narrowing down Safari bugs to minimal reproducible test cases, finding workarounds, and writing everything up in detail on the webkit bug tracker the only public response from Apple is the radar importer bot, and then everything goes quiet.
Btw, your original twitter thread implies you were the author of the bug report. Minor annoyance, but I did spend quite a while writing that up :)
ZapBox started as a Google Cardboard-inspired approach for video see-through MR, but we've now completely redesigned the physical product for the ZapBox use case.
The linked video shows a walkthrough of the older cardboard version, just because the tracking code for the new designs isn't production-ready yet.
Thought I'd share this with HN rather than the general Kickstarter campaign for the new design, as it goes into a bit more technical detail and shows some interesting issues in bringing stereo MR to existing smartphone devices.
Happy to discuss anything technical ZapBox-related here, relating either to the old or the new version. And of course if you want to back the current Kickstarter campaign, that'd be much appreciated too :)
Thanks for the note, I'll add some more caveats to that section; I don't really know anything about the human vision system. Do you know of any better mathematical models of human perception that I could use to produce more accurate mock-ups?
I've tried on some of the glasses from Meta and Epson that consist of more traditional micro-displays and optics to set a fixed virtual focus plane a couple of metres away. It's definitely possible to shift attention between the real and the virtual, and not feel too distracted even with relatively busy background scenes (using some of the "shades" definitely helps to boost contrast of the virtual content though). When real and virtual are at different focal planes it seemed to make maintaining attention on the virtual content easier, and stereo also seems to help with that.
However the true promise of "Mixed Reality" experiences are when you don't have to choose to attend to real or virtual, but simply look at a part of the combined scene. When virtual and real are so closely connection it seems likely to me that the depth of the virtual content would need to be matched with to the depth of the real world, so the virtual and real focal planes would be the same and the user would lose the ability to use depth cues to direct attention.
If real and virtual are at the same depth, how would you imagine the scene would look? Is the additive mock-up still unrealistic?
The video makes no claims of how small the tech is, but I agree it's unlikely to be in a consumer package at this stage. The display itself is mainly passive optical components. The input light can come from a single optical fibre from a pocket-sized smartphone-like battery/compute device. Scanning single-optical-fibre projection has been demonstrated by people now at Magic Leap - see the 1mm projection midway down this page:
http://gizmodo.com/how-magic-leap-is-secretly-creating-a-new...
Magic Leap have clearly put out mocked-up, offline composited videos before - they are clean, have perfect tracking, and do not use purely additive blending for the virtual content (they look much like Microsoft's HoloLens "live demos"). In both cases I have no doubt those are fake.
However this one definitely looks genuine to me - it's got all the hallmarks we'd expect from a scanning light-field display such as described in their patent application - in some frames it's possible to see two frames of virtual content in one frame of the video, suggesting a 60fps refresh of the content. The graphics on show are not beyond the capability of smartphone-type devices, and 3D depth cameras have been available in consumer elecronics since the first Kinect. I'd expect the demos are currently driven by desktop-type PC hardware, but there is nothing on show here that seems unrealistic to me.
Of course the watermark in the top right and the text at the bottom was composited afterwards but I'll let them get away with that.
This blog is my take on Microsoft HoloLens and how it compares to Magic Leap's proposition. I'll keep checking this thread and try to respond to any comments.
And from the video linked in the Gizmodo article, Graeme Devine claims they don't call it AR because "that is just a 2D HUD", and that's why they insist on the weird marketing phrase "Cinematic Reality".
To me and most people who have been around AR for years it is very clear that registration of objects in 3D is a core part of a lot of AR experiences. That was the part of the video that struck me as most weird; that Magic Leap would make a lot of noise around AR and yet have a Creative Content VP seemingly so unaware of the existing AR field.
In the land of "not really good enough". Additively mixed at a single focus depth plane, not dissimilar to the hacks people have been using in research for AR for a long time.
It does however really annoy me the amount of press Magic Leap have managed to get without saying anything public at all. For me it's an opportunity to think and talk about the properties of an ideal AR HMD and why it might be an interesting device. I have no idea how close Magic Leap are going to get to that ideal.
My main point is that making the ideal AR display is hard. To get anywhere close needs controllable opacity and virtual focus planes. From the patents that's definitely the direction Magic Leap are heading in.
If such a device exists and is 100% transparent when not in use then there are reasons why you might want to wear it all the time - think notifications like Google Glass but that can be embiggened and responded to on demand in a much larger portion of your field of view.
Early hardware is probably not quite going to be there in terms of portability, battery life, fashion, etc. So perhaps it makes sense to focus on in-home use cases first. However in the home-use scenario the value of AR is less clear; an Oculus planetarium may well be better than an AR one in my living room - I don't really want to limit the awesomeness of the size of the universe by the small size of my flat.
A lot of cases where AR adds value is in small experiences but deeply context-relevent ones. The consumer value equation for those experiences only really makes sense if the device is always-on. That's the future I think we're heading towards.
It's more about capturing the spectrum from AR to VR in a single device. With controllable opacity (which I argue is required for a truly useful AR display) you also have the option to deliver purely VR experiences by blocking out the real world completely.
To expand upon what the problem is: imagine rays of light reflecting from a point going in all directions (essentially a sphere). A cone of those rays make it into your eye through the pupil. Putting a 2D layer just near to the eye means light from that one point is actually spread across a circle on that 2D layer, with diameter close to the pupil size (getting on for 4mm in indoor lighting). There is therefore significant overlap between the circles from distinct 3D points - so to block out just one of the points you need to selectively block the rays over that entire circle, but only those in the direction of the point we wish to block.
A big unanswered question I have is what the level of transparency is going to be when the display is inactive. For me it needs to be basically 100% to enable an "always worn" piece of hardware. The patents around the lightfield waveguides suggest polarization is used to allow independent optical paths for rays from the real world vs rays coming from the virtual image. Again, I don't know much about optics so wonder if there's an electronic way to have that polarisation turned off for areas without any virtual content, or if there's always going to be a darkening effect on the real world.
Yeh, the controllable opacity is really important to the utility of the device IMHO.
I've yet to study the linked patent so can't say if that solves it but it's not as simple as just a layer of LCD pixels, as they would not be in focus.
It's sort of the same as the light-field generation problem - to generate images at arbitrary depths you need to control both the colour and direction of light rays. To generate sharp occlusions at arbitrary depths using a single 2D "masking" layer will require the layer selectively passing or blocking rays depending on their direction.
The patent mentions variable diffraction gratings, which I suspect might be part of the solution to controllable opacity based on ray directions, but I don't know enough about optics really to say for sure.
...distinct from volumetric displays which support multiple simultaneous viewers so need to actually render in terms of voxels.
Combining that with tracking depth cameras and sharing the map of the environment over the network would let multiple co-located users see and interact with the same 3D object (much like with a volumetric display) but the actual rendering of each user's view is decoupled.
Combining this with eye-tracking, as \hyp0 mentions, takes this to the limit - you don't need to construct a "holographic display" that renders super high resolution light fields of rays in controllable directions to all the infinite possible viewpoints around the display. Instead just stick it on your head, locate the users in a shared depth map, and just render the slice of that their eye needs to see right now. In terms of CPU/GPU/bandwidth issues it's much more plausible with today's technologies. The display side is still challenging!
In general you just to provide the depth map too. You're still only rendering for one viewpoint (well OK, one for each eye) so the display just needs to simulate the depth and colour of the point nearest to you along any ray.
We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon").
Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience.
The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement.
The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do.
What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :)
[1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.