I like you are having success here, and the key item "natural communication lines" which I very much agree with. We will always deliver conway's law.
I am curious on 1) the size of your teams, and 2) your opinion on how this would work with E2E product domain based teams (mixed skill sets) vs skills based teams e.g. User engagement team (front and backend devs covering sign on, emails, etc) vs Backend engineers (maintain all backend systems, etc).
If the browser uses WKWebView or UIWebView, you're right, you can "mock" an API by injecting JS, and through tunnelling you can recreate native functionality. Not if its SFSafariViewController. As someone who has done a lot of fiddling with JS to Obj-C/Java tunnelling for iOS and Android in production, it can be prone to errors and/or get your app rejected. Browser makers probably do not see the commercial value in going down this bespoke path.
Good question, still a "green" technology but I have had great successes with it for the past 4-5 years in production (ASM.JS before widespread WASM compatibility).
"promising that we'll be able to build web apps in any language" is not how I see WASM, nor is it really used in this way outside of transpiling Unity3D/Unreal games (this may be the one area there is an exception). I use it to transpile C++ to WASM libraries used within React apps, Edge Lambdas, and Node.JS servers. Primarily down to 2 reasons: speed and efficiency. WASM unlocks excellence and resources in other disciplines/languages such as AI and AR tool chains, Engineers, OpenCV, etc. When used like this, it is outstanding.
What WASM will never be good at is being used for the whole experience. You lose the semantic web, and/or accessibility tooling. Web has some outstanding guidelines and frameworks to help the impaired, screen readers and the like. Using WASM to pump a native app into a HTMLCanvasElement will lose all of these advances, therefore, WASM shouldn't be used for this use case (outside of games). Like all tooling, there is a time and a place to use them.
Below are a few links which use WASM in production:
iOS and Android don't have the same camera picker like Chrome on Windows 10, even then it is difficult to always override. I would recommend using enumerateDevices to allow you to pass the deviceId when firing getUserMedia. Not a lot of work. Implemented this a few times for cross-browser WebRTC experiences. A pro-tip is use getUserMedia with audio and video constraints set to true. This forces the permissions pop up to appear. You can kill the feed once the user has accepted the permissions, then ask them what device they want to use, and then fire getUserMedia with the correct deviceId. This is a [nice?] hack as some browsers don't allow you to use enumeratedDevices until after getUserMedia has asked you for the permissions. e.g. Safari...
Good question, I would say neither, this responsibility is for the browser and native apps you use which display web content (e.g. Chrome, Firefox, Facebook, etc). Not apple or the web app developer. At most OP should include https://www.npmjs.com/package/webrtc-adapter as part of their polyfills to aid cross-browser support/cater for WebRTC nuances.
WebRTC has been in iOS Safari for quite a while. However, all other apps and browsers were limited to what was in offer within WKWebView. From iOS 14.3 beta 1 onwards WKWebView gained getUserMedia/WebRTC features. It is now up to the browser and apps you use to make sure they have implemented their web views correctly and allow the correct permissions for WebRTC to work.
Used to live around the corner. Nice owner, good food and a great beer selection. I now [Ironically?] live in the states and miss that pub quite a lot.
To allow surgeons to interact with my software within an operating room without the need for an assistant (to remain sterile).
Interactive retail displays outside the store. Users can interact with augmented reality displays and visualize themselves wearing the store products and/or to play a game to win prizes, etc.
Problems encountered:
Hardware adequate for long experiences e.g. Microsoft Life camera freezes after a few hours. Finding a device which can run 24/7 is a problem. Then once you found a good device you need to understand the risk of it being pulled from the market e.g. Primesense, Kinect, Intel RealSense (pulled and replaced by a new product and SDK, etc).
If a depth camera is used the type of bulbs to sunlight can interfere with tracking accuracy. If RGB is solely used then I am curious to see how well it works with various skin-tone in different lighting conditions and complicated backgrounds.
The "heavy arm/hand" problem. Try lifting your hand for 5 minutes and not putting it down. Users can be fatigued very quickly with a gesture based UX. Most products are not designed for this interaction.
In terms of Zesture:
The website is clean, to the point, great starting point. However I would like to:
- See an Enterprise license for long term support
- Know how well it benchmarks against other SDKs/hardware solutions which achieve the same effect
- Patents, does this infringe on other proprietary innovations? (do you have patent troll insurance?)
- Guidelines for the best experience, e.g. distance from the camera if you were to use gestures to control a presentation
- Roadmap, where are you going next?(FYI I am looking for a new way of hand based gestures which can be deployed via WebRTC and WebAssembly for interactive web based experiences :) )
This is good advice. I would also extend this and write out an FAQ / stackexchange for the next engineer at your company who has to go through the same learning curve.
I do a significant amount of work with Hospitals and secure environments (Military, etc). TURN is needed 100% of the time. P2P traffic is not allowed. All IP addresses need to be known and kept static upfront for firewall whitelisting.
This means products which help alleviate WebRTC infrastructure such as AWS Kinesis are not allowed (due to how they allocate turn servers with unknown IP addresses) and a company needs to manage their own infrastructure / TURN servers (which allows you to cherry pick where server locations are (HIPAA, country legal for what is streamed)) or accept Twillio's, or their competitors etc, large IP ranges (and don't have server location flexibility / increased commercial and market growth restrictions).
Whichever route you go down it is quite an undertaking!
P.s. Tsahi Levent-Levi is truly exceptional in this area. I highly recommend reading his blog and training courses: https://bloggeek.me/, https://webrtccourse.com/, AND he runs an amazing testing product https://www.testrtc.com. if you build your own infrastructure testRTC is a must.
There are too many non constructive negative comments. Not many web engineers understand how the dom really works, or how to make a library. I applaud this tech demo.
Would I use this in production? No. But the author is not asking for that. It takes a lot of guts to create something and show it to this community. I really enjoyed looking at how the code progressed, how the author tackled a difficult subject, and their replies to constructive comments. Keep it up!
When dealing with Health or Military systems installing or updating a native application could result in months of delays (e.g. quarterly OS image update cycles). However running within Chrome, Firefox, and other typical software preinstalled, this becomes <days for implementation.
Without WebAssembly I wouldn't have been able to ship 2 products pro-bono within intensive care units and operating theatres directly helping with COVID.
I understand your dislike towards WebAssembly (albeit Web stack trends / flavour of the month esque development). I am not the largest fan of modern web development. Nevertheless love for WebAssembly is not due to developer productivity. After shipping 20+ WebAssembly products (alongside native counterparts) I am yet to meet an Engineer who enjoyed the WebAssembly/Emscripten/Blazor pipeline. However what WebAssembly has achieved for me is: Do people use your app? and within certain markets it allowed me to grow, do good, and say yes. This is the only real reason why someone should go down this route.
I think WebAssembly is more used than it appears, just difficult to see/tell.
A few years ago I actually tried integrating AR via WebAssembly with Amazon. We couldn't get the approval due to poor performance on Amazon fire devices (which have low end hardware). It is a shame but it is what it is.
What is disappointing/annoying is - as a CTO - it is near impossible to hire someone with WebAssembly skills. It requires an extra curious Engineer with a passion for native and web. Training is always important for a team but when going down the WebAssembly route you need to extra focused and invest more than what a typical Engineer would be allocated (E.g. Increase training from 1 day a week to 2-3). I suppose this may put people off?
Ten years ago I did the same but in Java and JOGL (before Apple banned OpenGL graphics within Java Applets embedded within a webpage). Was used for AR Watch try on within https://www.watchwarehouse.com and Ebay. The pain of Flash and Applets still wake me up at night.
I'm also building something very similar but with the ability for custom codecs (https://www.v-nova.com/ is very good). Probably the same issues too! Could I know more about your solution?
I use WebAssembly for a few cross-platform plugins. E.g. An AR 3D rendering engine in C++ and OpenGL. With very little effort it is working in browser. No bespoke code, same business logic, etc. Saved a lot of time vs creating a new renderer for our web app.
For me it allows a suite of curated plugins which work cross-platform. The web experience is nearly just as nice as the native mobile and desktop experience. This in turn increases market growth as more of my clients prefer web vs downloading an app (which is a large blocker for my users). I also enjoy the code reuse, maintainability, etc, :)
Another:
This year Max Factor (via Holition Beauty tech) won a Webby award for in-browser AI and AR. This was used to scan a users face, analyse their features, advise them on what make up, etc, would suit them, after which the user can try it on. This would have been impossible without WebAssembly.
This tech is also used by another makeup brands beauty advisors (via WebRTC) to call a customer and in real-time advise them on their make up look, etc.
Is this tech necessary? Probably not, but it is a lot nicer than having to go to a store. Especially when we are all in lockdown :)
I understand your point but I see this as a global B2B requirement. Most products I use and have built cover North America, Europe, Middle East and now Africa. Each country or region has its quirks. If you are GDPR compliant there is a good chance you cover most other regulation. Nevertheless is an issue, especially - and rightly so - the fines are increasing and the regulation is becoming tighter. If it is not built in the extendability of a product to do so does matter and therefore the success of the template.
yes. I am a data processor not a controller. If you store data tied to the user you still need a mechanism to retrieve / delete it. This is something I encountered. Even if it is in reality not an issue and by itself the data is meaningless. Perhaps I need better legal advice :)
Love this, OpenCV and all WebAssembly projects. I also use next-translate now and then so kudos for that!
I have been putting AR in-browser when Java applets with JOGL was a thing! I've been nominated twice this year for the Webby awards on AI and AR in browser (1). Small innovative team who have been utilising Emscripten and likeminded technologies for a few years from when Emscripten and WebRTC was starting to be a thing.
I wanted to share some pain points taking this tech to production.
- Bandwidth
This is huge with OpenCV, ~4.5Mb+ to take a picture is quite a difficult bandwidth cost to accept. Especially the clients I worked with have millions of views per day. The total binary for Max Factor VMUA (2) is the same size which includes a large data set needed for a neural network for skin tone analysis and face feature detection.
Learning: Do not include all of OpenCV. You don't need it all, but if you do cherry pick the parts you need. I do recommend writing the simplistic parts (this is for you who just use cv::mat!).
- Speed
If you want a 60 FPS AR effect / AI algo on an Android device OpenCV isn't always the fastest approach. Do not rely on a framework, you will need to get your hands dirty and optimise/rewrite the slow areas. WebAssembly is fast, but not as fast as the desktops and native environments you normally create this code on.
- Market
Not everyone has an iPhone in London. Bandwidth means seconds, JS and WebAssembly execution adds to this. In a world where m-commerce is king this does matter. Think Poland, middle of nowhere in Ohio, Brazil, etc. If it takes 60 seconds for a web app to run on 3g and then another 20 for the executable to start, and then the experience is then sluggish it wont be commercially successful.
- UX
When you put this into a large site most traffic will come via instagram and facebook. On iOS this is typically within a WKWebView which does not support getUserMedia. Make sure you have some nice hints on how to open within iOS Safari (or Android Chrome if the parent app has not enabled permissions).
Nevertheless I wish this blog post existed when I started out. I regret in not writing something similar. In this post I especially love the simplicity of the Emcripten pipeline which is great. It is a fantastic demo post. I do hope it inspires many to play with this innovative stack.
I like this, I see what they're doing and it looks promising. However It lacks a data request/GDPR takedown section which for me makes the project commercially useless.
Most teams/products always look at compliance as the last big rock to implement and it becomes a massive headache. If this project is focused to B2B learning it should have been tackled first (and not missing from the demo).
I am curious on 1) the size of your teams, and 2) your opinion on how this would work with E2E product domain based teams (mixed skill sets) vs skills based teams e.g. User engagement team (front and backend devs covering sign on, emails, etc) vs Backend engineers (maintain all backend systems, etc).