Introducing TensorFlow.js: Machine Learning in JavaScript(medium.com)
medium.com
Introducing TensorFlow.js: Machine Learning in JavaScript
https://medium.com/tensorflow/introducing-tensorflow-js-machine-learning-in-javascript-bf3eab376db
45 comments
recurrentjs https://github.com/karpathy/recurrentjs
convnetjs https://cs.stanford.edu/people/karpathy/convnetjs/
I think the author of these also had one for reinforcement learning.
convnetjs https://cs.stanford.edu/people/karpathy/convnetjs/
I think the author of these also had one for reinforcement learning.
brain.js if I remember correctly is a very old project and is no longer maintained. But it was great to learn the basic ANN concepts. What I would love to see is a Node.js implementation of deep learning and ML libs (those listed above are targeted for browsers). With Node.js supporting most of es6+ features it's been a fun coding in it. Can't wait to get hands on ML with server side JavaScript as well..
It is maintained now: http://brain.js.org/
...I'm the maintainer
...I'm the maintainer
It seems to have been updated quite a bit. With support for more modern NN architectures. But I don't know how fast it is relative to the others.
I think my favorite part of this is that inevitably people will write type definitions for TensorFlow.js in TypeScript, Flow, ReasonML, Purescript, and more.
It has always seemed like data science and machine learning tasks have always been most popular in dynamic languages like Python, Julia, and R. I really hope this can be a bridge over to typed machine learning APIs!
(Of course, I’m also interested to hear about your favorite typed machine learning library if you have some that you already use)
It has always seemed like data science and machine learning tasks have always been most popular in dynamic languages like Python, Julia, and R. I really hope this can be a bridge over to typed machine learning APIs!
(Of course, I’m also interested to hear about your favorite typed machine learning library if you have some that you already use)
> I think my favorite part of this is that inevitably people will write type definitions for TensorFlow.js in TypeScript
Actually, it already ships with typescript definitions - it's written in TypeScript![1]
[1] https://github.com/tensorflow/tfjs/blob/master/src/index.ts
Actually, it already ships with typescript definitions - it's written in TypeScript![1]
[1] https://github.com/tensorflow/tfjs/blob/master/src/index.ts
Absolutely spot on... I love watching Siraj Raval's videos on ML - but darn it every other video has a 2 or 3 minute segment where he's just scratching his head - finally to realize there is a typo that could have been caught with a typed language!
"favorite typed machine learning library"
Flux is essentially nothing but some clever use of Julia's type system:
https://github.com/FluxML/Flux.jl
There's some ability to generate javascript too, which you can try here: http://fluxml.ai/experiments/mnist/ . Edit: I see it's actually using deeplearn.js == TensorFlow.js for this, details at https://github.com/FluxML/FluxJS.jl
Flux is essentially nothing but some clever use of Julia's type system:
https://github.com/FluxML/Flux.jl
There's some ability to generate javascript too, which you can try here: http://fluxml.ai/experiments/mnist/ . Edit: I see it's actually using deeplearn.js == TensorFlow.js for this, details at https://github.com/FluxML/FluxJS.jl
I am starting to really like Javascript now that Typescript showed up. Makes things so much nicer.
Neural net layers should have been statically typed ages ago, as anyone who has struggled with mismatching array sizes in Python already knows.
I'm really looking forward to a superior dev experience with writing ML code once great autocompletion kicks in
I'm really looking forward to a superior dev experience with writing ML code once great autocompletion kicks in
I would love to see some full Tensor Flow bindings for Go. It's concurrency system is great for data science tasks. The statically linked binary makes it great for distributed workflows.
How about TensorFlow for Swift? :-D
https://news.ycombinator.com/item?id=16719615
https://news.ycombinator.com/item?id=16719615
I think dynamic languages are a bad fit, because they force the programmer to think in data-flow graphs, whereas a more static language could allow for more offloading of complexity to a compiler (i.e. let the compiler build the data-flow graph, which is essentially what we've been doing for decades).
A good set of Haskell bindings to the TensorFlow library:
https://github.com/tensorflow/haskell
https://github.com/tensorflow/haskell
For folks who want to immediately play around with this — I've ported over the sentiment analysis example to an Observable notebook, so you can just edit the code directly:
https://beta.observablehq.com/@jashkenas/sentiment-analysis-...
... or just have fun watching it analyze random Metacritic reviews of ”The Last Jedi”.
https://beta.observablehq.com/@jashkenas/sentiment-analysis-...
... or just have fun watching it analyze random Metacritic reviews of ”The Last Jedi”.
Thanks for this! As a non-ML'er (for lack of a better term), I enjoyed seeing how simple it could be to port this kind of amazing stuff into a JS app :)
Very nice.
Here is my review:
"This is very good as in a very good cancer. It is as good as a dirt burger."
Positivity 0.9992088079452515
"This is very good as in a very good cancer. It is as good as a dirt burger."
Positivity 0.9992088079452515
This is really cool. I was wondering what the difference between this and deeplearnjs[0] were, and turns out it's the same thing. Between this, and propelml[1] (from the creator of node.js), it seems like ML in the JS ecosystem is starting to pick up.
It'd be cool to see more demos like the ones on the https://tenso.rs/ website. Seems like they were using their own hand rolled library though.
[0]: https://deeplearnjs.org/ [1]: https://github.com/propelml/propel
It'd be cool to see more demos like the ones on the https://tenso.rs/ website. Seems like they were using their own hand rolled library though.
[0]: https://deeplearnjs.org/ [1]: https://github.com/propelml/propel
deeplearn.js has been rebranded as tensorflow.js
There are already some cools demos built with tensorflow.js(https://deeplearnjs.org/demos/performance_rnn or https://teachablemachine.withgoogle.com)
There's also a high-level library built on top of tensorflow.js that makes it even easier to use some common ML methods: https://github.com/ml5js/ml5-library (disclosure: I'm part of the team developing it)
There are already some cools demos built with tensorflow.js(https://deeplearnjs.org/demos/performance_rnn or https://teachablemachine.withgoogle.com)
There's also a high-level library built on top of tensorflow.js that makes it even easier to use some common ML methods: https://github.com/ml5js/ml5-library (disclosure: I'm part of the team developing it)
Those sites aren't working on google chrome Mac Version 65.0.3325.162 (Official Build) (64-bit)
I don't know what the issue is it tells me to download chrome.
I don't know what the issue is it tells me to download chrome.
If you are interested in using production-grade ML from JS -- and the full data science and big data eco-system -- a bunch of us in the Apache Foundation and GPU world have been joining under Arrow and GOAI: https://devblogs.nvidia.com/goai-open-gpu-accelerated-data-a... .
The Graphistry team and ccri have been investing on zero-copy nodejs bindings to the full ecosystem, eg, leverage full TenserFlow, AWS GPUs, CuDNN, SciPy, pandas, dask, multigpu, MapD, etc. If you're a node hacker, this is an open effort -- contributors welcome! You can reach me at leo@[ graphistry], and I'll be publishing some JS roadmaps summarizing recent meetings and release directions.
The Graphistry team and ccri have been investing on zero-copy nodejs bindings to the full ecosystem, eg, leverage full TenserFlow, AWS GPUs, CuDNN, SciPy, pandas, dask, multigpu, MapD, etc. If you're a node hacker, this is an open effort -- contributors welcome! You can reach me at leo@[ graphistry], and I'll be publishing some JS roadmaps summarizing recent meetings and release directions.
Dumb question: Does this:
"TensorFlow.js automatically supports WebGL, and will accelerate your code behind the scenes when a GPU is available."
imply that you can get all the benefits of GPU-accelerated machine learning on any old GPU---not just CUDA-compatible Nvidia GPUs?
For example, if you've got one of those 15 inch Macbook pros with a GPU, can you use TensorFlow.js to train a model at comparable (even if not identical) speed to a custom deep learning box?
"TensorFlow.js automatically supports WebGL, and will accelerate your code behind the scenes when a GPU is available."
imply that you can get all the benefits of GPU-accelerated machine learning on any old GPU---not just CUDA-compatible Nvidia GPUs?
For example, if you've got one of those 15 inch Macbook pros with a GPU, can you use TensorFlow.js to train a model at comparable (even if not identical) speed to a custom deep learning box?
Yes, it will work on almost any GPU (even mobile!). No, it will not be anywhere near the speed of CUDA.
But I guess it will be as fast or faster than native CPU code.
cool, thank you!
This looks very cool.
Does the WebGL integration mean it doesn't require installation of CUDA or cuDNN on the machine it's running on? That's currently the most difficult part by far of getting GPU-enabled TensorFlow up and running in Python.
Additionally, can it work with non-NVIDIA GPUs? That is another fairly stringent current limitation for those that don't want to cough up the dough for an NVIDIA card or pay for cloud service.
Does the WebGL integration mean it doesn't require installation of CUDA or cuDNN on the machine it's running on? That's currently the most difficult part by far of getting GPU-enabled TensorFlow up and running in Python.
Additionally, can it work with non-NVIDIA GPUs? That is another fairly stringent current limitation for those that don't want to cough up the dough for an NVIDIA card or pay for cloud service.
It does not require CUDA or cuDNN on the client machine, it should work on most GPUs (even mobile) but won't be as performant as CUDA / cuDNN.
why ppl want to run ML in the browser ? Sorry I can not think of a use case.
One important reason is that it helps democratize ML to make it accessible to everybody. Even a casual hobby programmer can write awesome applications, and anybody - especially non-technical users - can use them. It makes ML - especially inference - as easy as opening facebook.
Right now, just to consume ML models - whether as a application developer or an end user - requires some combination of special skills that fall in a spectrum of complexities - from something relatively simple like installing a system package or an environment like Anaconda or a pip package, to something much more complex and time consuming like building TF or Caffe.
ML in browser bypasses all of that.
Right now, just to consume ML models - whether as a application developer or an end user - requires some combination of special skills that fall in a spectrum of complexities - from something relatively simple like installing a system package or an environment like Anaconda or a pip package, to something much more complex and time consuming like building TF or Caffe.
ML in browser bypasses all of that.
I still don't understand -- do you have an example application in mind?
You seem to be conflating producing ML models -- i.e. doing machine learning -- with "consuming" ML models -- i.e. asking the learned models to make a prediction. You don't need any ML in the browser to do the latter. And I can't see why you'd do ML in the browser to do the former...
You seem to be conflating producing ML models -- i.e. doing machine learning -- with "consuming" ML models -- i.e. asking the learned models to make a prediction. You don't need any ML in the browser to do the latter. And I can't see why you'd do ML in the browser to do the former...
[Update: Oh boy, I realized this is a gigantically long post after posting it. Sorry about that, but I hope my step by step explanation convinces you.]
Not conflating - the benefits apply equally to both learning and inference, but since there are magnitudes more potential consumers of inference than learning, I emphasized it.
It's true that one doesn't need any of this, but my point is not having them in browser means there are barriers put up - of complexity, of costs, of privacy, of effort - to developers and end users.
I'll use face recognition as a walkthrough example, but this applies to absolutely any ML use case if you think about all the steps involved in taking it from idea to development to deployment to end use.
Take a problem I've worked on a bit - intelligently searching through personal photos and videos. Most people have atleast a few hundreds of GBs of photos and videos - family photos, pets, travels - in aggregate across all their devices. Some may feel the need for search software that can answer questions like "find me that photo with Alice (user's daughter) playing with Scooby(user's dog) from 10 years ago".
In a world without browser ML, how would a developer design, develop and deploy this with maximum convenience for both development and end use? Maybe like this...
- dev starts off by deciding they don't want to mess with any of that ML stuff. Their skills lie in front-end design and usability. They decide to go with Amazon's or Google's face recognition service (IDK if AMZ/GOO actually have such a service, but if they did, it's reasonable a dev would look at them as the first option).
- But they soon find out it's just shifting the complexity elsewhere. Now, they have to provide a way for users to upload their hundreds of GBs of media to S3 or GCS. Which means more APIs to learn and integrate. More costs for storage. Usability barriers and privacy suspicions for users. Security aspects have to be looked into. Looks like it'll have to become a paid service now.
- The service by itself is not enough. Dev still has to provide the front-end (which they are skilled at) for users to select photos, crop faces, apply labels, and send it all to the service's transfer learning API.
- After all that, some users complain that accuracy is not good enough because it couldn't find many photos. Dev has no way to tweak the models because those are behind another company's opaque service. It's increasingly looking like a custom backend is necessary.
- So version 2. Dev learns some ML. Then downloads a pre-trained model that can do face detection and recognition - say FaceNet or OpenFace.
- They have to deploy it server-side for training and indexing. They learn a bit of Nginx and WSGI, and deploy it. They don't know how many users will use and how much data will be uploaded - have to plan automated scaling for that. EC2 or GCE? More stuff and more APIs to learn, and more costs.
- Dev still has to provide the front-end for users to select photos, crop faces, apply labels, and upload to their learning service. Dev has to implement per-user transfer learning and store per-user transfer data and models.
- Dev has to implement all the required provisioning for inference and transfer learning - be it raw GPU servers or docker or K8s or whatever. More costs.
- For an end user, the need to upload hundreds of GBs of personal media to a 3rd party is also a barrier - takes time, loses privacy and likely incurs bandwidth costs.
- So version 3. Dev says forget the server-side. User already has GBs of photos in their hard disks. Instead of bringing their photos to us and managing it, let's take the software to them. Let's just package up everything and allow user to download and use the entire thing on their local machines. Maybe as platform-specific installables. Or as platform-neutral docker image. Reduces costs and complexity for developer. Can even be free since there are no costs incurred by developer. Android's still a problem since it can't do docker, and dev doesn't know Android app development.
- The end user too benefits with far better privacy and usability. However, they still have to install a package - sounds easy, but in a world of "user does not have administrative privileges" and "sudo", there are still potential barriers to cross. And Android is still a no-go because the dev doesn't know it.
Now in a world with browser ML, you can see how those remaining problems too can be solved. Javascript ML is write once, run on any browser - even Android's. User does not have to install anything. Dev does not have to write anything specifically for a different platform. All the transfer learning and inference can happen in user's browser.
The browser environment still presents some barriers - such as not being able to access local photos directly without user selecting them, and limited local storage for models. But both can be solved with some creative batching and using solutions like emscripten's virtual file system in memory (I'm not sure if TF.js uses the latter, but other frameworks like OpenCV.js do). User pays some cost of reduced usability, which they may be ok with since they may see the alternative options as being worse. And the privacy is matchless.
All this is applicable to any ML use case. Anything involving user's private data such as speech recognition or document scanning/OCR too get the exact same benefits for both developers and users.
Not conflating - the benefits apply equally to both learning and inference, but since there are magnitudes more potential consumers of inference than learning, I emphasized it.
It's true that one doesn't need any of this, but my point is not having them in browser means there are barriers put up - of complexity, of costs, of privacy, of effort - to developers and end users.
I'll use face recognition as a walkthrough example, but this applies to absolutely any ML use case if you think about all the steps involved in taking it from idea to development to deployment to end use.
Take a problem I've worked on a bit - intelligently searching through personal photos and videos. Most people have atleast a few hundreds of GBs of photos and videos - family photos, pets, travels - in aggregate across all their devices. Some may feel the need for search software that can answer questions like "find me that photo with Alice (user's daughter) playing with Scooby(user's dog) from 10 years ago".
In a world without browser ML, how would a developer design, develop and deploy this with maximum convenience for both development and end use? Maybe like this...
- dev starts off by deciding they don't want to mess with any of that ML stuff. Their skills lie in front-end design and usability. They decide to go with Amazon's or Google's face recognition service (IDK if AMZ/GOO actually have such a service, but if they did, it's reasonable a dev would look at them as the first option).
- But they soon find out it's just shifting the complexity elsewhere. Now, they have to provide a way for users to upload their hundreds of GBs of media to S3 or GCS. Which means more APIs to learn and integrate. More costs for storage. Usability barriers and privacy suspicions for users. Security aspects have to be looked into. Looks like it'll have to become a paid service now.
- The service by itself is not enough. Dev still has to provide the front-end (which they are skilled at) for users to select photos, crop faces, apply labels, and send it all to the service's transfer learning API.
- After all that, some users complain that accuracy is not good enough because it couldn't find many photos. Dev has no way to tweak the models because those are behind another company's opaque service. It's increasingly looking like a custom backend is necessary.
- So version 2. Dev learns some ML. Then downloads a pre-trained model that can do face detection and recognition - say FaceNet or OpenFace.
- They have to deploy it server-side for training and indexing. They learn a bit of Nginx and WSGI, and deploy it. They don't know how many users will use and how much data will be uploaded - have to plan automated scaling for that. EC2 or GCE? More stuff and more APIs to learn, and more costs.
- Dev still has to provide the front-end for users to select photos, crop faces, apply labels, and upload to their learning service. Dev has to implement per-user transfer learning and store per-user transfer data and models.
- Dev has to implement all the required provisioning for inference and transfer learning - be it raw GPU servers or docker or K8s or whatever. More costs.
- For an end user, the need to upload hundreds of GBs of personal media to a 3rd party is also a barrier - takes time, loses privacy and likely incurs bandwidth costs.
- So version 3. Dev says forget the server-side. User already has GBs of photos in their hard disks. Instead of bringing their photos to us and managing it, let's take the software to them. Let's just package up everything and allow user to download and use the entire thing on their local machines. Maybe as platform-specific installables. Or as platform-neutral docker image. Reduces costs and complexity for developer. Can even be free since there are no costs incurred by developer. Android's still a problem since it can't do docker, and dev doesn't know Android app development.
- The end user too benefits with far better privacy and usability. However, they still have to install a package - sounds easy, but in a world of "user does not have administrative privileges" and "sudo", there are still potential barriers to cross. And Android is still a no-go because the dev doesn't know it.
Now in a world with browser ML, you can see how those remaining problems too can be solved. Javascript ML is write once, run on any browser - even Android's. User does not have to install anything. Dev does not have to write anything specifically for a different platform. All the transfer learning and inference can happen in user's browser.
The browser environment still presents some barriers - such as not being able to access local photos directly without user selecting them, and limited local storage for models. But both can be solved with some creative batching and using solutions like emscripten's virtual file system in memory (I'm not sure if TF.js uses the latter, but other frameworks like OpenCV.js do). User pays some cost of reduced usability, which they may be ok with since they may see the alternative options as being worse. And the privacy is matchless.
All this is applicable to any ML use case. Anything involving user's private data such as speech recognition or document scanning/OCR too get the exact same benefits for both developers and users.
Sorry for the late response, but I want to thank you for the in-depth post! I agree with you that version 3 is way better, but I'm very cautious about advocating for browsers as de-facto operating systems. If we want better cross-platform systems for sandboxing and running programs, I'd prefer to develop those directly instead of giving more power to browsers and browser vendors.
how about real time object detection/facial rec/semantic segmentation via webcam
what are some options for consuming ML models in the browser?
Let's not exaggerate; using something like pip is incredibly easy compared to actually using the ML package. The hardest thing about ML is not setting it up...
It's platform independent. Most deep learning libraries have absurd dependencies. They are difficult to install and only support specific OSes and GPUs. Javascript will run literally anywhere.
Think more from the inference angle. There are a lot of use cases for ML and many are using native apps or desktop apps. This allows you to run it on the browser on the web, which significantly broadens the amount of people who can use it.
So can you think of use cases for using ML in an app (native or desktop?) Many of those would be good to have in the browser as part of a web app too.
So can you think of use cases for using ML in an app (native or desktop?) Many of those would be good to have in the browser as part of a web app too.
Become more creative.
For everyone's reference, I had remembered there was already a library that existed that claimed to do this.
I was digging around to find it, it's called deeplearn.js, and upon checking it looks like that project has officially moved to the one referenced in this very post.
https://deeplearnjs.org/
I was digging around to find it, it's called deeplearn.js, and upon checking it looks like that project has officially moved to the one referenced in this very post.
https://deeplearnjs.org/
For anyone interested in using ML in JavaScript, checkout this status quo article [0] about it. Tensorflow.js has to be added to it still.
- [0] https://www.robinwieruch.de/machine-learning-javascript-web-...
- [0] https://www.robinwieruch.de/machine-learning-javascript-web-...
Super cool, I hope it will help getting more people around ML + bring better ergonomics to ML overall.
[tensorflow.js](https://github.com/tensorflow/tfjs)
[mxnet.js](https://github.com/dmlc/mxnet.js/)
[propel.js](https://github.com/propelml/propel) - this uses tensorflow.js under the hood
[ml5](https://github.com/ml5js/ml5-library) - also uses tensorflow.js
[webdnn](https://github.com/mil-tokyo/webdnn)
[brain.js](https://github.com/BrainJS/brain.js)
[mljs](https://github.com/mljs/ml)
[synaptic](https://github.com/cazala/synaptic)
edit: added the ml5 lib mentioned in a subsequent comment.