Maybe I didn't make the point well in the article. It is very limited to special cases when they limit the IBIS pixel shift to in-camera processing with only a JPEG output.
The camera should take pictures with the IBIS fractional picture shift and save the RAW files. They should give the option of how many "cycles" (go through all the shift orientations more than once) of pictures to take. With that level of information, smart software will be able to figure out and deal with at least small hand motions and considerable motion in the subject.
Smartphones are already using computational photography, combining multiple photos for things like panning and taking pictures in the dark. For a dedicated camera like the R5 mk ii, I would want the camera to save RAW images that can be put together later by "smart" software on a computer with greater processing and memory resources.
Good catch on the "combined with custom catadioptric lenses." Which by definition means a combination of mirrors and refractive optics. But I still don't think they are pancake lenses, but something more akin to what Limbak, which was recently bought by Apple was famous for designing catadioptric optics. Before being bought by Apple, Limbak was best known for their catadioptric design used by Lynx.
You are correct that MR today implies that the Virtual image is locked to the real world with some form of SLAM.
I was a bit sloppy in that regards. I was more worried about what was going on. But when combined with the word "passthrough" to see the real world, I tend to drop back/slip to the term AR passthrough which is what this type of thing was called for years. MR, as I remember it, is a more recent term used to distinguish different types of AR. Then things flipped and XR (=AR/VR/MR) was used to mean was AR used to mean.
My guess is that late in the program, the importance of passthrough was elevated, perhaps in response to Apple rumors, but they were stuck with the hardware that was in the pipeline. The passthrough available is a big improvement over the Quest 2 for use in seeing the grouse-level surroundings, but not up to regular use in full Mixed Reality.
There are probably decades of "optimizations" left to do.
If you are talking for many "game playing" applications you may be correct.
But it be ridiculous to thing that people are going to be going out in public with a VR headset with cameras (known as Passthrough AR). On top of a number of human factor issues, it would dangerously block a person's view of the real world.
Yes, I took the picture of Avegant used in the article. The Avegant and Dispelix waveguide combination looks pretty good. It is only a prototype without any tracking/slam. It is a display only demonstration by a component company.
I was impressed by the size of the Avegant engine and the transparency and lack of forward projection by the Dispelix prototype. They are claiming they will get 2,000 nits to the eye out of the design which should be good enough for outdoor use IF they have some form of clip-on sunglasses (2,000 nits is not enough for outdoors in full sunlight without some help).
The current Avegant engine has an optical component in that was depolarizing the light from the LCOS display and losing contrast so they wanted me to wait to take through the lens pictures. The image from the current prototype looked sharp but did lack contrast.
I don't think the issue is color. Their original product was trying to differentiate on Vergence Accommodation Conflict (VAC) and did a poor job of it. They were also going after the consumer with a product that was always going to be way too expensive for consumers with an image quality that was going to be too poor. They also made what I think is a bad set of trade-offs in terms of ergonomics and human factors.
The problem with medicine is that is it a tiny market. You can't justify the type of money they were trying to raise for that market. Under Abovitz, they were always a swing for the fences company.
All the above said, nobody I know of is making money selling AR headsets today. They are either subsidized by VC or other investment money or by big companies funding R&D efforts. The market is not other product areas where you can make money with an MVP and grow the product.
While the market is "hot" in terms of awareness, it is still a very small market facing major technical challenges. I think AR while useful in enterprise markets (measured in the 100's of thousands a year) is a long way from being ready for the mass consumer market.
I am seeing a lot of progress in some areas and will be publishing an article later this week on them. In particular both Dispelix and Digilens have made considerable progress on the "glowing eyes" issues (Dispelix all but eliminates it). Avegant has a very nice small LCOS light engine that pairs nicely with the Dispelix waveguide.
I think the biggest problem for AR is that the expectations are very high and the physics is very tough. Many physical optical features within a few wavelengths of light were diffraction ruins everything.
They seems to have done some good things. Sadly in AR as well as many high tech markets, it is possible to have a great technical achievement but miss the market requirements. The dimming feature caused them to lose at least 70% of the real-world light off the top. Nothing else they could do would made up for that mistake. It looks like they were focused on novel features over utility.
I think a large part of this is that they were building the ML2 for the high end consumer video game market when one day they they were told it was too expensive and it would now be an enterprise product. I hate the display quality of the HL2, but at least they improved ergonomic and interface issues over the HL1, whereas the ML2 repeats the mistakes of the ML1.
The rumor mill is that Microsoft has slowed down development of HL3 development with many people leaving. Most famously, Bernard Kress, and optical architect of Hololens 1 and 2 has left for Google.
There were very few Magic Leap Ones ever sold. The number I hear is between 5,000 and 10,000 units sold for money with about as many units given away.
Hololens 2 seems to be shipping about 70K to 100K units per year but it is not clear if that has peaked or not. The display quality is very poor as I have written about extensively, but the ergonomics are good enough for some industrial applications.
I get the sense that ML2 was too far gone and with money running out to react to ML2 and thus still has terrible ergonomics. With blocking 80% of the light, not supporting ordinary glasses, and having a cable which is a snag hazard, the future looks bleak for Magic Leap 2 as it was for ML1. I don't know if they would have been a success, but at least they would have had a shot in the same space as HL2 if they had reworked the serious ergonomic problems. The dimming feature is a differentiation that resulted in them missing the market altogether.
As others have said, that is probably not the best use of Tilt-5's technology. No matter how you move your head, it is still 1280x768 pixels spread over 110 degrees.
Doing a little math, that is only about 13 pixels per degree or 4.5 arcminute/pixel. Typical VR headsets are about 30 pixels/degree (2-arcminutes per pixel. For optical text reading like with a computer monitor, I consider 40 pixels/degree (1.5arcminutes/pixel) the bare minimum.
There is also some scintillation from the beaded screen. Not terrible, but enough that it would not be good for text. So even if the resolution was higher (which is possible with this technology), I'm still not sure it would be good as a computer monitor.
The case where it works best seems to be in "tabletop" applications, and it behaves best when the viewer is about 45 degrees to the surface. Business and military applications for things such as "sand tables" also fit this model.
The camera should take pictures with the IBIS fractional picture shift and save the RAW files. They should give the option of how many "cycles" (go through all the shift orientations more than once) of pictures to take. With that level of information, smart software will be able to figure out and deal with at least small hand motions and considerable motion in the subject.
Smartphones are already using computational photography, combining multiple photos for things like panning and taking pictures in the dark. For a dedicated camera like the R5 mk ii, I would want the camera to save RAW images that can be put together later by "smart" software on a computer with greater processing and memory resources.