I like you are having success here, and the key item "natural communication lines" which I very much agree with. We will always deliver conway's law.
I am curious on 1) the size of your teams, and 2) your opinion on how this would work with E2E product domain based teams (mixed skill sets) vs skills based teams e.g. User engagement team (front and backend devs covering sign on, emails, etc) vs Backend engineers (maintain all backend systems, etc).
If the browser uses WKWebView or UIWebView, you're right, you can "mock" an API by injecting JS, and through tunnelling you can recreate native functionality. Not if its SFSafariViewController. As someone who has done a lot of fiddling with JS to Obj-C/Java tunnelling for iOS and Android in production, it can be prone to errors and/or get your app rejected. Browser makers probably do not see the commercial value in going down this bespoke path.
> If the browser uses WKWebView or UIWebView, you're right, you can "mock" an API by injecting JS, and through tunnelling you can recreate native functionality.
So, technically it can be done.
What's stopping them? Do you think Apple would count that sort of feature as "implementing your own web engine"?
Good question, still a "green" technology but I have had great successes with it for the past 4-5 years in production (ASM.JS before widespread WASM compatibility).
"promising that we'll be able to build web apps in any language" is not how I see WASM, nor is it really used in this way outside of transpiling Unity3D/Unreal games (this may be the one area there is an exception). I use it to transpile C++ to WASM libraries used within React apps, Edge Lambdas, and Node.JS servers. Primarily down to 2 reasons: speed and efficiency. WASM unlocks excellence and resources in other disciplines/languages such as AI and AR tool chains, Engineers, OpenCV, etc. When used like this, it is outstanding.
What WASM will never be good at is being used for the whole experience. You lose the semantic web, and/or accessibility tooling. Web has some outstanding guidelines and frameworks to help the impaired, screen readers and the like. Using WASM to pump a native app into a HTMLCanvasElement will lose all of these advances, therefore, WASM shouldn't be used for this use case (outside of games). Like all tooling, there is a time and a place to use them.
Below are a few links which use WASM in production:
> "promising that we'll be able to build web apps in any language" is not how I see WASM
Unfortunately it kind of got branded this way for people who aren't close to the front-end industry. There's a segment out there who would like to write web apps but aren't willing to touch JavaScript with a ten-foot pole, and their hopes were gotten up that WASM would give them that. It's not exactly a lie, but it has so many asterisks that it may as well be.
In practice, the reality is that JS will continue to be the only first-class language for the web. WASM has many uses, some on the web and some outside of it, but the closer your app is to the DOM and to the browser as a platform, the less likely it is that you'll ever be able to pretend JS doesn't exist.
Blazor client-side (for dotnet) works fine with minimal asterisks, and even those issues are being slowly solved as WASM evolves with multithreading, GC, and DOM access.
> What WASM will never be good at is being used for the whole experience.
The Flutter team would disagree.
They are leaning on WASM for browser builds of Flutter apps, with the whole app rendering in a canvas.
They do accessibility via separately created accessibility trees.
The experience far from great at the moment, but give it a few years and I think it will get there. (better wasm optimisations, direct host interop without JS shims, GC, threads, maybe WGPU instead of canvas, ...)
If that's a good thing for the web is another question...
Unfortunately, Flutter apps still seem to be lagging on the accessibility front. The Flutter web gallery[0] has many examples of beautiful apps, but something as "simple" (on the web) as being able to select text is absent.
For some use-cases, this is a nonstarter. (Sadly, not to as many as I'd like! But let's simply not regress, to start.)
It’s worth keeping in mind that Flutter for web had its 1.0 release a matter of weeks ago. It’s very impressive with what they have done so far but let’s see where it goes. The amount of potential there is truly huge.
It’s genuinely hard to tell if you are serious about this or not. Either way it somehow perfectly encapsulated the kind of thing this website is famous for. Congratulations?
Flutter feels so strange to me. In many ways, it's reimplementing major parts of a browser, compiled to (web)assembly, running inside a browser compiled to assembly... Watch five years from now someone add a low-level programming framework to Flutter and then soneone else reimplement the browser using that
Turtles all the way down! It would be more humorous if we used Skia, since that arguably fulfill's OPs proposition that someone would eventually rewrite the browser in said framework
The flutter web text field don't even handle multiline text input with IME properly. Cause the whole text field to scroll while select character from IME dropdown. I'd doubt how could that be count as "good user experience".
WebAssembly Summit 2021 was this week, and that is definitely not how the people on the frontline are selling it, BBC is even on the process to fully rewrite they iPlayer in C++.
iOS and Android don't have the same camera picker like Chrome on Windows 10, even then it is difficult to always override. I would recommend using enumerateDevices to allow you to pass the deviceId when firing getUserMedia. Not a lot of work. Implemented this a few times for cross-browser WebRTC experiences. A pro-tip is use getUserMedia with audio and video constraints set to true. This forces the permissions pop up to appear. You can kill the feed once the user has accepted the permissions, then ask them what device they want to use, and then fire getUserMedia with the correct deviceId. This is a [nice?] hack as some browsers don't allow you to use enumeratedDevices until after getUserMedia has asked you for the permissions. e.g. Safari...
Good question, I would say neither, this responsibility is for the browser and native apps you use which display web content (e.g. Chrome, Firefox, Facebook, etc). Not apple or the web app developer. At most OP should include https://www.npmjs.com/package/webrtc-adapter as part of their polyfills to aid cross-browser support/cater for WebRTC nuances.
WebRTC has been in iOS Safari for quite a while. However, all other apps and browsers were limited to what was in offer within WKWebView. From iOS 14.3 beta 1 onwards WKWebView gained getUserMedia/WebRTC features. It is now up to the browser and apps you use to make sure they have implemented their web views correctly and allow the correct permissions for WebRTC to work.
Used to live around the corner. Nice owner, good food and a great beer selection. I now [Ironically?] live in the states and miss that pub quite a lot.
To allow surgeons to interact with my software within an operating room without the need for an assistant (to remain sterile).
Interactive retail displays outside the store. Users can interact with augmented reality displays and visualize themselves wearing the store products and/or to play a game to win prizes, etc.
Problems encountered:
Hardware adequate for long experiences e.g. Microsoft Life camera freezes after a few hours. Finding a device which can run 24/7 is a problem. Then once you found a good device you need to understand the risk of it being pulled from the market e.g. Primesense, Kinect, Intel RealSense (pulled and replaced by a new product and SDK, etc).
If a depth camera is used the type of bulbs to sunlight can interfere with tracking accuracy. If RGB is solely used then I am curious to see how well it works with various skin-tone in different lighting conditions and complicated backgrounds.
The "heavy arm/hand" problem. Try lifting your hand for 5 minutes and not putting it down. Users can be fatigued very quickly with a gesture based UX. Most products are not designed for this interaction.
In terms of Zesture:
The website is clean, to the point, great starting point. However I would like to:
- See an Enterprise license for long term support
- Know how well it benchmarks against other SDKs/hardware solutions which achieve the same effect
- Patents, does this infringe on other proprietary innovations? (do you have patent troll insurance?)
- Guidelines for the best experience, e.g. distance from the camera if you were to use gestures to control a presentation
- Roadmap, where are you going next?(FYI I am looking for a new way of hand based gestures which can be deployed via WebRTC and WebAssembly for interactive web based experiences :) )
This is good advice. I would also extend this and write out an FAQ / stackexchange for the next engineer at your company who has to go through the same learning curve.
I do a significant amount of work with Hospitals and secure environments (Military, etc). TURN is needed 100% of the time. P2P traffic is not allowed. All IP addresses need to be known and kept static upfront for firewall whitelisting.
This means products which help alleviate WebRTC infrastructure such as AWS Kinesis are not allowed (due to how they allocate turn servers with unknown IP addresses) and a company needs to manage their own infrastructure / TURN servers (which allows you to cherry pick where server locations are (HIPAA, country legal for what is streamed)) or accept Twillio's, or their competitors etc, large IP ranges (and don't have server location flexibility / increased commercial and market growth restrictions).
Whichever route you go down it is quite an undertaking!
P.s. Tsahi Levent-Levi is truly exceptional in this area. I highly recommend reading his blog and training courses: https://bloggeek.me/, https://webrtccourse.com/, AND he runs an amazing testing product https://www.testrtc.com. if you build your own infrastructure testRTC is a must.
STUN is only useful if you're trying to negotiate a P2P connection, which isn't the case when using an SFU. If everything you're doing is going through an SFU then you don't need STUN.
I am curious on 1) the size of your teams, and 2) your opinion on how this would work with E2E product domain based teams (mixed skill sets) vs skills based teams e.g. User engagement team (front and backend devs covering sign on, emails, etc) vs Backend engineers (maintain all backend systems, etc).