AR with SceneKit and Metal(emillindfors.com)
emillindfors.com
AR with SceneKit and Metal
https://emillindfors.com/blog/2020-12/ar-with-scenekit-and-metal/
45 comments
Yeah I think it needs killer hardware before a killer app.
They're laying the platform groundwork early which makes sense since it's a bet on what's next for human computer interaction.
If they can pull the glasses off there's huge opportunity there for all sorts of things which will be really interesting and better than looking at little glass screens. Without that hardware it's super niche and I don't think has much broad use.
If anyone can pull it off when the hardware is ready it'll probably be Apple.
FB is obviously trying really hard to be there too, but I just don't think people will want it to come from FB (and forcing Oculus under their brand makes it so that it would have to be).
Magic Leap will be to whatever Apple ends up making as General Magic was to the iPhone.
The rough idea was directionally correct, but way too early with hardware that just wasn't good enough.
They're laying the platform groundwork early which makes sense since it's a bet on what's next for human computer interaction.
If they can pull the glasses off there's huge opportunity there for all sorts of things which will be really interesting and better than looking at little glass screens. Without that hardware it's super niche and I don't think has much broad use.
If anyone can pull it off when the hardware is ready it'll probably be Apple.
FB is obviously trying really hard to be there too, but I just don't think people will want it to come from FB (and forcing Oculus under their brand makes it so that it would have to be).
Magic Leap will be to whatever Apple ends up making as General Magic was to the iPhone.
The rough idea was directionally correct, but way too early with hardware that just wasn't good enough.
I don't think most people "care" it's from Facebook, as someone who bought the Oculus Quest 2 and thought about a long time before that to buy the valve index.
The same could have been said about the chromecast.
I think price matters a lot and Apple already loses a big audience because of that ( it's not their market, i know).
By reference, the IPad is still priced reasonable cheap.
Ps. The Oculus works well with minimal setup.
The same could have been said about the chromecast.
I think price matters a lot and Apple already loses a big audience because of that ( it's not their market, i know).
By reference, the IPad is still priced reasonable cheap.
Ps. The Oculus works well with minimal setup.
Yeah you’re probably right.
Facebook’s other big disadvantage is that their platform (oculus) is separate from mobile and Apple’s platform (iOS) isn’t.
This makes it a lot easier for a Apple to leverage existing devices in laying platform groundwork (which they have now been doing for years).
It also makes it easier for them to leverage the phone computing for the glasses hardware similarly to how they did that for the watch.
FB doesn’t have this option really and I’m not sure how they’d do it without it. Try and put all the hardware in the headset itself? Seems less likely to be able to ship first or be as nice as what Apple will be able to do.
Facebook’s other big disadvantage is that their platform (oculus) is separate from mobile and Apple’s platform (iOS) isn’t.
This makes it a lot easier for a Apple to leverage existing devices in laying platform groundwork (which they have now been doing for years).
It also makes it easier for them to leverage the phone computing for the glasses hardware similarly to how they did that for the watch.
FB doesn’t have this option really and I’m not sure how they’d do it without it. Try and put all the hardware in the headset itself? Seems less likely to be able to ship first or be as nice as what Apple will be able to do.
Totally agree, AR in games is more of a toy. The progress with lidar is... progress - but still not great even in basic scenes, helps with low light. Visual issues break the immersion and pull the user from the experience, which is supposed to be the point. AR as a tech is interesting, and has practical business uses as you point out - but has a long way to go to be useful for games, let alone an engaging centerpiece. Like everything else, it’ll improve steadily, and I’m excited for where it could go
Agree.
One of the most interesting AR games I’ve tried was Kings of Pool same-space multiplayer. So you set up a pool table (=billiard?) and walk around the room around the same table. Worked quite well, the game (AR & network) could keep up with the relatively low tempo of a turn-based game well enough. But that was mostly interesting because of novelty, not necessarily sticky for long-term repeat play.
What prompted me to explore the tech and write the blog post was indeed work on something else than a game.
One of the most interesting AR games I’ve tried was Kings of Pool same-space multiplayer. So you set up a pool table (=billiard?) and walk around the room around the same table. Worked quite well, the game (AR & network) could keep up with the relatively low tempo of a turn-based game well enough. But that was mostly interesting because of novelty, not necessarily sticky for long-term repeat play.
What prompted me to explore the tech and write the blog post was indeed work on something else than a game.
From what I observed, the killer AR game so far was the one involving Pokémons.
Even though people rarely play it while on a sofa, would it be workable with the current Apple AR hardware?
Even though people rarely play it while on a sofa, would it be workable with the current Apple AR hardware?
I never played the game myself, but wasn't AR just a decorative add-on to pokemonGo and not a necessary part of the game?
Not necessary at all, and one of the first things I turned off as it was distracting and would sometimes force you into weird positions in order to catch a pokemon.
Doesn't LIDAR solve these technical problems?
It will once it rolls out onto more phones. Right now it is only available on iPhone 12 Pro models
Where the use-case currently is more photogrammetry
https://en.wikipedia.org/wiki/Photogrammetry
https://en.wikipedia.org/wiki/Photogrammetry
Very cool,
I've been starting to use ARKit from a different angle.
Usecase - test out 3d designs of replacement classic car parts I'm manufacturing as a side project.
While crude right now, I think this sort of stuff is a game changer - I'm regretting not picking up a lidar enabled 12 pro instead of a 12 now to do better scanning and placement. I guess I'll pick up an ipad pro when it's upgrade time on that front.
Here are two quick videos I threw together:
https://youtu.be/M8dgrf7GBow
https://www.youtube.com/watch?v=7K0-vK2wafA
Workflow:
- build 2d models in solvespace (https://solvespace.com)
- export dxf to fusion 360
- convert dxf to sheet metal
- bend sheet metal in fustion360
- export obj from fusion360
- save obj file to icloud drive
- open obj file in AR Viewer app (https://apps.apple.com/us/app/ar-viewer-augmented-reality/id...)
Sidenote:
There still isn't a replacement for physically printing the parts before I have them laser cut out of metal. I 3d print models and test fit pieces as intermediate steps too.
Really excited to see ios developers digging into ARKit more, I'm excited to see what else is possible here.
I've been starting to use ARKit from a different angle.
Usecase - test out 3d designs of replacement classic car parts I'm manufacturing as a side project.
While crude right now, I think this sort of stuff is a game changer - I'm regretting not picking up a lidar enabled 12 pro instead of a 12 now to do better scanning and placement. I guess I'll pick up an ipad pro when it's upgrade time on that front.
Here are two quick videos I threw together:
https://youtu.be/M8dgrf7GBow
https://www.youtube.com/watch?v=7K0-vK2wafA
Workflow:
- build 2d models in solvespace (https://solvespace.com)
- export dxf to fusion 360
- convert dxf to sheet metal
- bend sheet metal in fustion360
- export obj from fusion360
- save obj file to icloud drive
- open obj file in AR Viewer app (https://apps.apple.com/us/app/ar-viewer-augmented-reality/id...)
Sidenote:
There still isn't a replacement for physically printing the parts before I have them laser cut out of metal. I 3d print models and test fit pieces as intermediate steps too.
Really excited to see ios developers digging into ARKit more, I'm excited to see what else is possible here.
If you got the iPhone after November 10th they have an extended holiday return period https://www.apple.com/shop/help/returns_refund
Yep, I'm in that window, but availability of the 12 pro is pretty bad and it's not that painful in the grand scheme of life to warrant hunting one down haha. Regret, but not that big of a regret :)
Well you've got another month to think about it.
Personally LIDAR on the phone sounds a lot more useful to me than the iPad, and I say that having a 2020 iPad Pro.
Knew I was going to upgrade my SE to a 12 Mini with no LIDAR (based on rumors) so I sprung for the new iPad instead of looking for 2018s on sale. Nice to have one device with LIDAR to play around with, but it's super awkward to wave an iPad around to scan things.
More than anything else, I think it was pushed out as a developer kit to get the hardware in people's hands and apps ready to go before it hit the mass market with the iPhone 12 Pro.
Personally LIDAR on the phone sounds a lot more useful to me than the iPad, and I say that having a 2020 iPad Pro.
Knew I was going to upgrade my SE to a 12 Mini with no LIDAR (based on rumors) so I sprung for the new iPad instead of looking for 2018s on sale. Nice to have one device with LIDAR to play around with, but it's super awkward to wave an iPad around to scan things.
More than anything else, I think it was pushed out as a developer kit to get the hardware in people's hands and apps ready to go before it hit the mass market with the iPhone 12 Pro.
Not sure if it'll work for you but I was successful in getting a same-day courier for the 12 Pro for $9 where shipping it would have taken a month.
Thanks for sharing, hadn’t considered something like this, very interesting to see!
Would be cool if the object-matching tech (this[1] + more machine learning?) in ARKit was advanced enough that it could recognize a car seat or dashboard and place things automagically.
[1] https://developer.apple.com/documentation/arkit/scanning_and...
Would be cool if the object-matching tech (this[1] + more machine learning?) in ARKit was advanced enough that it could recognize a car seat or dashboard and place things automagically.
[1] https://developer.apple.com/documentation/arkit/scanning_and...
How are you enjoying solvespace? I wanted to use it for a project but I found that I was completely unable to use it for integer-parametric designs. E.g. if I wanted 10 of something stacked end-to-end, I couldn't figure out how to do that without breaking the solver.
I like it more for 2d work.
I haven’t gotten into the groove with it for 3d workflows. I probably need to do some more tutorials for it on that front.
I haven’t gotten into the groove with it for 3d workflows. I probably need to do some more tutorials for it on that front.
Pretty cool application. Exactly how accurate is the AR + Lidar combo when it comes to fitting? For example, if you 3D print a car part after sizing it in 3D, does the printed product match up?
I haven't been able to scan the interior (transmission tunnel in this case).
But that would be the general idea. Scan interior - 3d model parts to fit in that scanned world - 3d print/CNC cut those parts to fit.
But that would be the general idea. Scan interior - 3d model parts to fit in that scanned world - 3d print/CNC cut those parts to fit.
I highly recommend watching the anime Dennou Coil [0] to see what the world could be like if AR glasses were as common as smartphones today:
• Everybody is connected to the same "AR space" and sees the same AR objects.
• Different people may have different view/interact access to different objects.
• Some form of tactile feedback is assumed. In Apple's case we will assume this to be provided via the Apple Watch or a hypothetical AR gloves accessory.
• Many things that are currently physical could be replaced with virtual versions: screens, keyboards, signage, fashion, pets.
Many developers have already noticed the gradual introduction of APIs and hardware like LIDAR that are so obviously meant to support a future glasses product, but I wonder if anyone's connected the dots between features like the spatial audio recently introduced to AirPods.
Apple's AR strategy is very likely going to be divided across their different products: iPhone (processing), Glasses (display), Watch (motion control and tactile feedback), and of course AirPods. Many of their users already own most of those devices. No other company is so perfectly positioned to finally deliver ubiquitous AR.
[0] https://myanimelist.net/anime/2164/Dennou_Coil
• Everybody is connected to the same "AR space" and sees the same AR objects.
• Different people may have different view/interact access to different objects.
• Some form of tactile feedback is assumed. In Apple's case we will assume this to be provided via the Apple Watch or a hypothetical AR gloves accessory.
• Many things that are currently physical could be replaced with virtual versions: screens, keyboards, signage, fashion, pets.
Many developers have already noticed the gradual introduction of APIs and hardware like LIDAR that are so obviously meant to support a future glasses product, but I wonder if anyone's connected the dots between features like the spatial audio recently introduced to AirPods.
Apple's AR strategy is very likely going to be divided across their different products: iPhone (processing), Glasses (display), Watch (motion control and tactile feedback), and of course AirPods. Many of their users already own most of those devices. No other company is so perfectly positioned to finally deliver ubiquitous AR.
[0] https://myanimelist.net/anime/2164/Dennou_Coil
The interesting bit about Dennou Coil is the one corporation that essentially controls a significant portion of the protagonists' lives ;)
Also endless data collection of literally everything in every persons FOV, ads aggressively plastered over everything in your life at all times, subtle censorship of anything corporations or government would rather you not see, etc.
As with every technology we have right now it will be used against you. It doesn't have to be that way, but that's the world we live in now.
As with every technology we have right now it will be used against you. It doesn't have to be that way, but that's the world we live in now.
I think AR kit is the most under-utilized API available on iOS. Combined with the U1 chip, you could make a real-world tagging-app. Post a note on your favorite trail or maybe overlay In n Out's secret menu on their real menu. How about AR Zoom backgrounds?
I think you're spot on. The documentation for AR Kit is top notch, and includes TONS of sample code that is easily understandable with minimal iOS development knowledge.
Perhaps I'm still too new to AR Kit, but to me it doesn't look entirely straightforward (more backend lifting) to do someone of the things you are mentioning. I hope Apple releases some updated Geolocation API's to better enable the experiences you mention.
Kai Faust's [1] work with Infopop [2] is definitely worth playing around with
[1]: (https://twitter.com/kaifaust)
[2]: (https://apps.apple.com/us/app/infopop/id1530246692)
Perhaps I'm still too new to AR Kit, but to me it doesn't look entirely straightforward (more backend lifting) to do someone of the things you are mentioning. I hope Apple releases some updated Geolocation API's to better enable the experiences you mention.
Kai Faust's [1] work with Infopop [2] is definitely worth playing around with
[1]: (https://twitter.com/kaifaust)
[2]: (https://apps.apple.com/us/app/infopop/id1530246692)
The building blocks for location based AR are pretty much all there:
- Azure Spatial Anchors (iOS, Android, HoloLens, ROS) [1]
- Google Cloud Anchors (iOS, Android) [2]
- Apple Geo Anchors (iOS & not all locations so not ready for use yet) [3]
Integration with these SDKs is pretty easy, and Azure Spatial Anchors and Google Cloud Anchors are production-ready. Minecraft Earth for example used Azure Spatial Anchors for shared multiplayer AR experiences. Infopop is nice, but doesn't have multiplayer or long-term persistence, so not really useful as-is. But wouldn't be surprised if it eventually uses one of these SDKs to provide better multiplayer + shared relocalization. You could probably throw together a rough version of Infopop in a week if sufficiently motivated, ARKit does a lot of the heavy lifting there.
[1]: (https://docs.microsoft.com/en-us/azure/spatial-anchors/overv...)
[2]: (https://developers.google.com/ar/develop/java/cloud-anchors/...)
[3]: (https://developer.apple.com/documentation/arkit/argeoanchor)
- Azure Spatial Anchors (iOS, Android, HoloLens, ROS) [1]
- Google Cloud Anchors (iOS, Android) [2]
- Apple Geo Anchors (iOS & not all locations so not ready for use yet) [3]
Integration with these SDKs is pretty easy, and Azure Spatial Anchors and Google Cloud Anchors are production-ready. Minecraft Earth for example used Azure Spatial Anchors for shared multiplayer AR experiences. Infopop is nice, but doesn't have multiplayer or long-term persistence, so not really useful as-is. But wouldn't be surprised if it eventually uses one of these SDKs to provide better multiplayer + shared relocalization. You could probably throw together a rough version of Infopop in a week if sufficiently motivated, ARKit does a lot of the heavy lifting there.
[1]: (https://docs.microsoft.com/en-us/azure/spatial-anchors/overv...)
[2]: (https://developers.google.com/ar/develop/java/cloud-anchors/...)
[3]: (https://developer.apple.com/documentation/arkit/argeoanchor)
Thanks for sharing the links to Google Cloud Anchors and Azure! Quickly looked through the docs of both, excited to try them out and see how well they work :)
I used Apple's Geo Anchors, but found it pretty limiting for my use case (labelling rock climbs outdoors), since it rely's heavily on street level data it seems
I used Apple's Geo Anchors, but found it pretty limiting for my use case (labelling rock climbs outdoors), since it rely's heavily on street level data it seems
Yep, they're both pretty comparable in their API surface, and more general purpose than Apple Geo Anchors, as neither are tightly coupled with GPS. Rather you create an anchor and then save it to the cloud. On your own you can tie it to a GPS location if you'd like. Though Azure Spatial Anchors does have more functionality than Google Cloud Anchors (Coarse Relocalization + find nearby anchors in the anchor graph).
William Gibson predicted those use cases in his 1994 novel Virtual Light. He had artists creating works of art in physical locations that were only visible to people wearing AR goggles.
https://www.penguinrandomhouse.com/books/59659/virtual-light...
https://www.penguinrandomhouse.com/books/59659/virtual-light...
I haven't tried it myself, but Reality Mixer is a neat application of person segmentation that I saw recently: https://twitter.com/reality_mixer
Impressive how well that works.
Impressive how well that works.
Thanks for sharing, good to know someone has built this!
Clearly, we need a ‘Wikiworld’ AR app for adding shared metadata to the world.
Unity has become a strong platform choice for AR development as it wraps both ARKit and ARCore and is cross-platform with iPhones and Android phones, as well as webgl. Pokemon Go, probably the AR app with the most market penetration, is Unity.
Agree, in most cases Unity is probably a better tool for AR application work than using the platform APIs directly.
[deleted]
Great article and great conversation here. I wonder if you have some good communities where you spend time. I wish I could read much more of these. I created a forum but so far I have been the only one posting https://augmented-reality-forum.tribe.so/post/hello-world-5f...
Hi lankinen, maybe try some big AR subreddits like /r/augmentedreality, /r/AR_MR_XR or /r/virtuality. Could also google for some slack or discord channels.
I don’t necessarily frequent specific communities too much, more just try to find stuff via googling about various interests and projects, and talking to friends and colleagues.
I don’t necessarily frequent specific communities too much, more just try to find stuff via googling about various interests and projects, and talking to friends and colleagues.
The HoloDevelopers Slack is very active. HoloLens focused, but also channels for other AR topics: https://holodevelopersslack.azurewebsites.net/
nice post, worth knowing though that I have heard through the grapevine that SceneKit is to be deprecated in favor of RealityKit. SceneKit also has abhorrent documentation that is just plain wrong in places.
Seems logical that it would happen at some point. I don’t think RealityKit covers enough of the semi-common use cases in its current state that they could deprecate SceneKit just yet. But it’s on the horizon definitely.
I'll be waiting for an AR app to overlay billboards with 'OBEY', 'CONSUME', etc. personally.
Is SceneKit still alive? I had the idea that it was abandoned by Apple.
Making a game where people have to sit in front of a table, or stand, or hold the device up, is very restrictive and instantly makes you niche. You get more of a potential slice of the market making a cool 2D puzzle.
The tech was cool, worked reasonably well until -A- you moved a lot -B- low light. But unfortunately the market is about playing on the toilet, in public transport; or playing late at night in bed, in the dark. None of which really worked with AR.
As far as I see this going, it's still is looking for a "killer app" (I hate that term).
My best guess is we'll see this used a lot more in a business environment and even then, glasses are a far better approach (hint hint, hold on to your whiskers).