Meta Releases New AI-Based Photo Segmentation Tool to Everybody(arstechnica.com)
arstechnica.com
Meta Releases New AI-Based Photo Segmentation Tool to Everybody
https://arstechnica.com/information-technology/2023/04/meta-introduces-ai-model-that-can-isolate-and-mask-objects-within-images/
81 comments
Discussion from yesterday: https://news.ycombinator.com/item?id=35455566
I relly hope Meta (FB research) releases this wonderful noise reduction library: https://github.com/facebookresearch/denoiser
I tested MANY while building various audio tech and this was by far the best (beats the shit out of all the python Pandas Hugging Face lists etc). Incredibly ability to cut noise (tho, it must be noted that perceptual improvements for humans do not usually increase machine transcription as AI models strangely seem to pull info out of the dead space between words, and the noise around words...not just the voiced words themselves...the hidden vibrations, beyond our human ken, oh mere mortals unworthy of the grand perceptive machines...ugh...:p :o ;p xx ;p)
I contacted the authors via their FB emails but never heard back. Right now it's non-commercial and I was building a commercial product.
I tested MANY while building various audio tech and this was by far the best (beats the shit out of all the python Pandas Hugging Face lists etc). Incredibly ability to cut noise (tho, it must be noted that perceptual improvements for humans do not usually increase machine transcription as AI models strangely seem to pull info out of the dead space between words, and the noise around words...not just the voiced words themselves...the hidden vibrations, beyond our human ken, oh mere mortals unworthy of the grand perceptive machines...ugh...:p :o ;p xx ;p)
I contacted the authors via their FB emails but never heard back. Right now it's non-commercial and I was building a commercial product.
Somewhat tangential question, have you looked for/found any audio models/tools that can be used for separating out individual voices to separate audio tracks automatically? Perhaps this is already possible with existing tools that I am uninitiated in.
That's called "speaker diarization" and there's quite a bit of work in the field. https://github.com/topics/speaker-diarization
I have no idea what's good or the best, but there's a starting point!
I have no idea what's good or the best, but there's a starting point!
Thank you for the proper terminology, googling these things in plain English just doesn't do it anymore
Not a problem and good luck!
I had speaker diarization and my wife and dog wouldn’t even come near me for weeks
Does Ultimate Vocal Remover 5 fit the bill?
Despite the name, it can also do audio separation.
Despite the name, it can also do audio separation.
meta has demucs which is the best I've used so far
I haven't tested this with multiple voices and it sounds like you want something more specific but it's produced 10/10 results with a couple dozen audio files I've thrown at it, might be of use... https://vocalremover.org/
Izotope RX Pro, which is software for the cleaning and refinement of audio for music and audio post production includes 'Multiple Speaker Detection' which analyzes different voices in a recording and allows you to process them independently.
https://www.izotope.com/en/products/rx.html
I can't speak to it's effectiveness because I don't have any need for it, and also RX 10 Advanced is commercial software and pretty expensive for a casual user, but the feature seems to be on the horizon for other apps.
https://www.izotope.com/en/products/rx.html
I can't speak to it's effectiveness because I don't have any need for it, and also RX 10 Advanced is commercial software and pretty expensive for a casual user, but the feature seems to be on the horizon for other apps.
Have you tested https://hushaudioapp.com ? Its quality is amazing.
How does it compare to Adobe's recent AI audio enhancement?
Not sure. Anyone else wanna chime in on that?
BTW - in my search just now in response to your question, I see Dolby also has an API: https://dolby.io/products/enhance/
BTW - in my search just now in response to your question, I see Dolby also has an API: https://dolby.io/products/enhance/
Interesting that all of these seem to work extremely well but none are actually available to end users as an unrestricted product.
Meta does stellar work in AI. I’m quite certain the recent DMCAs were a case of the left hand not knowing what the right hand is doing. This model is open source in every sense of the word, and they’ve now released two impactful models in the span of a month.
This tweet chain is a pretty great overview: https://twitter.com/drjimfan/status/1643647849824161792?s=61...
This tweet chain is a pretty great overview: https://twitter.com/drjimfan/status/1643647849824161792?s=61...
> left hand not knowing what the right hand is doing.
No it was trying to stop abuse. Unlike OpenAI, facebook can't get away with releasing a thing that is more than capable of making up libellous, racist or any other type of illegal or PR decimating thing.
The point of the really restrictive release was to at least try to limit that kind of issue.
Fortunatly for meta, most people who use the weights aren't the end user with a large twitter following, so the risk is (now) low.
No it was trying to stop abuse. Unlike OpenAI, facebook can't get away with releasing a thing that is more than capable of making up libellous, racist or any other type of illegal or PR decimating thing.
The point of the really restrictive release was to at least try to limit that kind of issue.
Fortunatly for meta, most people who use the weights aren't the end user with a large twitter following, so the risk is (now) low.
The Economist had a recent overview of the AI competitive landscape that highlights Meta's outsize role thanks to their embrace of openness. https://www.economist.com/business/2023/03/26/big-tech-and-t...
They're well on their way to yahoo'ing themselves - valuable and significant software contributions for a platform nobody is using.
I agree, Meta's work in AI has been really impressive, and it's encouraging how they are open-sourcing so much.
It's funny, back in the 90s I disliked Microsoft and thought they were inherently culturally opposed to open source. But big corporations really just follow their business incentives when it comes to open source.
It's not like Meta "loves to be open". But they would hate a world where all the powerful AI development happened on one of the big clouds. Imagine if every researchers used some closed-source API from Google or Azure or AWS. Meta would try to hire some AI people, and they'd be like, ugh if we take the job we have to use this weird Facebook-specific thing. (<cough>Flow</cough>)
So, supporting PyTorch and providing open source models is just a good business strategy for Meta. I'm glad to see it.
It's funny, back in the 90s I disliked Microsoft and thought they were inherently culturally opposed to open source. But big corporations really just follow their business incentives when it comes to open source.
It's not like Meta "loves to be open". But they would hate a world where all the powerful AI development happened on one of the big clouds. Imagine if every researchers used some closed-source API from Google or Azure or AWS. Meta would try to hire some AI people, and they'd be like, ugh if we take the job we have to use this weird Facebook-specific thing. (<cough>Flow</cough>)
So, supporting PyTorch and providing open source models is just a good business strategy for Meta. I'm glad to see it.
I’m glad Microsoft is open sourcing a lot of things.
I have never, ever seen a convincing business case for their doing so, however.
Not your opinion. A compelling business case.
I have never, ever seen a convincing business case for their doing so, however.
Not your opinion. A compelling business case.
It's pretty simple, they make a lot of money off Azure, and open source software is a complement to Azure. See the classic:
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
In particular, open source software prevents people from getting locked into any of the AWS, Linux, iOS, or Android ecosystems. All of those are popular software development ecosystems that Microsoft does not own or influence very much.
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
In particular, open source software prevents people from getting locked into any of the AWS, Linux, iOS, or Android ecosystems. All of those are popular software development ecosystems that Microsoft does not own or influence very much.
Not convincing or even relevant IMHO. Azure infra is largely closed source and Microsoft is making a ton of money off it.
The business case for open sourcing dev tooling is that more developers will use them and build on them and that brings more people to your platforms.
Sure. Any proof at all for that? I mean it's the obvious answer, but generally in business people who don't pay for things tend not to pay for things. In fact, in the enterprise having a big budget is the way you keep your corporate power. Free stuff isn't a way to keep your budget high.
Again, I would like to agree with your thinking. I just don't think the real world backs it up.
Again, I would like to agree with your thinking. I just don't think the real world backs it up.
> I agree, Meta's work in AI has been really impressive, and it's encouraging how they are open-sourcing so much.
My usual reaction to anything Facebook releases is "yawn, they released yet another model that is practically useless because it's released under a non-commercial license", but I'm pleasantly surprised that this one seems to be actually liberally licensed! Hopefully this continues into the future.
My usual reaction to anything Facebook releases is "yawn, they released yet another model that is practically useless because it's released under a non-commercial license", but I'm pleasantly surprised that this one seems to be actually liberally licensed! Hopefully this continues into the future.
[deleted]
Are the weights open source?
For anyone curious about file sizes for the PyTorch models:
* default/sam_vit_h_4b8939.pth: 2.4GB
* sam_vit_l_0b3195.pth: 1.2GB
* sam_vit_b_01ec64.pth: 358MB
* default/sam_vit_h_4b8939.pth: 2.4GB
* sam_vit_l_0b3195.pth: 1.2GB
* sam_vit_b_01ec64.pth: 358MB
Yes
Where? From the article:
>Currently, the code (without the weights) is available on GitHub
>Currently, the code (without the weights) is available on GitHub
The weights are linked from the readme; avoiding large binary resources in GitHub repositories is fairly normal.
In the README on said github
https://github.com/facebookresearch/segment-anything#model-c...
https://github.com/facebookresearch/segment-anything#model-c...
[deleted]
The weights are not "source".
That is likely true in the most pedantic sense, but in practice, if I create an algorithm that works by using a series of matrix transformations against a set of carefully chosen (read: "trained") matrices and I open-source only the matrix manipulation code but not the specially chosen matrices, I think there's a fair argument to say that I haven't open-sourced the entire algorithm.
In the phrase "the model is open-source in every sense of the word", that, IMO, must include the weights.
In the phrase "the model is open-source in every sense of the word", that, IMO, must include the weights.
I'd be curious to know if you think open source companies should share all of their database records too.
I think of two broad categories of database records: transactional data (data created while running the system) and domain data (data created during development and shipped to production as part of the release process).
The former type of data I wouldn't expect to ever be open-sourced. The latter type might or might not be, depending on the intent of open-sourcing the related system.
If I created a human language translation system that used a SQL database to store the dictionary (domain) data and claimed the system was open-source without shipping the domain data, I think people would rightly say that the system was not fully opened.
The former type of data I wouldn't expect to ever be open-sourced. The latter type might or might not be, depending on the intent of open-sourcing the related system.
If I created a human language translation system that used a SQL database to store the dictionary (domain) data and claimed the system was open-source without shipping the domain data, I think people would rightly say that the system was not fully opened.
The actual source code is not important. The source code can be printed on a single A4 page, the valuable final product is the weights you get after running the code for fifty million dollars of compute time.
If they start claiming they're "open weights" or "open final product" I'll be up in arms.
They are claiming that. They are also delivering on that claim: https://github.com/facebookresearch/segment-anything#model-c...
The weights seem to be under the same license, just distributed separately because it doesn’t make sense for the giant binary artifacts of training to be part of the source repository.
If I autogenerated a huge amount of C based on the weights, that added/multipled variables the same way the existing code+weights does, then would it be “source”?
[deleted]
I'm extremely sorry but I might have submitted a mismatch title and URL. I stumbled on about 5-6 outlets via Twitter.
Here is the announcement from Meta (erstwhile Facebook)
https://ai.facebook.com/blog/segment-anything-foundation-mod...
Here is the announcement from Meta (erstwhile Facebook)
https://ai.facebook.com/blog/segment-anything-foundation-mod...
I've been wanting to build an app that will transform a photo into a template for stained glass. Interactive, to choose the "granule" size, merge color groups into one single piece, change border thickness, etc. Something like this would really help there and make it almost a piece of cake.
Yeah it basically does the first part that with a few lines of code! I've been playing with it using it to segment images and just randomly shake each segment in a short gif. Just to kind of mess around with it until I've found a real use for it or more time to implement some of the more interesting use cases I've got. I'm super pumped this was released!
Perhaps it's not as sophisticated as Meta, but doesn't iOS16 do this?
I recall my partner discovering it by accident whilst messing about with some photos. The people in the foreground were lifted off the background.
I recall my partner discovering it by accident whilst messing about with some photos. The people in the foreground were lifted off the background.
It’s similar but the iOS just segment the main object in an image.
Yes, but it is closed to Apple's app only. Android has something something similar, but they haven't released to the public.
Meta is releasing it to everyone, so you can use it on your own apps/services. (and it is a great implementation/state of the art).
Meta is releasing it to everyone, so you can use it on your own apps/services. (and it is a great implementation/state of the art).
Thanks for confirming (wasn't sure if I'd misremembered it somehow!!)
And totally with you that the Meta release is great for getting this kind of thing more widely adopted.
And totally with you that the Meta release is great for getting this kind of thing more widely adopted.
Here's a great summary:
https://twitter.com/DrJimFan/status/1643647849824161792
https://twitter.com/DrJimFan/status/1643647849824161792
[deleted]
these are toy examples
If course they are. I haven't looked into this model in particular, but unless I'm way off base, these general purpose pretrained things are completely unsuitable for any specialized (say medical or whatever) application. This seems just like a better version of some of the previous semantic segmentation models they trained on coco.
It's probably a good starting point for fine tuning for custom application though, that's the main use of these big models.
It's probably a good starting point for fine tuning for custom application though, that's the main use of these big models.
Any idea how/when this will make it into Photoshop?
According to the article, the weights haven't been open sourced, so nobody can take this and build their own Photoshop plugin.
But I'm incredibly curious if they'll license this to Adobe, or if Adobe's already got its own team building the same thing that'll be just as good, or what.
It's not super-clear to me why Meta as opposed to Adobe has come out with this first, except maybe to be able to apply filters to just segments of photos in Insta?
According to the article, the weights haven't been open sourced, so nobody can take this and build their own Photoshop plugin.
But I'm incredibly curious if they'll license this to Adobe, or if Adobe's already got its own team building the same thing that'll be just as good, or what.
It's not super-clear to me why Meta as opposed to Adobe has come out with this first, except maybe to be able to apply filters to just segments of photos in Insta?
> According to the article, the weights haven't been open sourced, so nobody can take this and build their own Photoshop plugin.
The article seems to be misleading, since I can just go to the github and download the weights right now: https://github.com/facebookresearch/segment-anything#model-c...
However, for sake of fairness, competition, and reproducible science, I hope that if Adobe makes this a part of their commercial offering, they train their own so we can have two variants of the same model available to us.
The article seems to be misleading, since I can just go to the github and download the weights right now: https://github.com/facebookresearch/segment-anything#model-c...
However, for sake of fairness, competition, and reproducible science, I hope that if Adobe makes this a part of their commercial offering, they train their own so we can have two variants of the same model available to us.
Photoshop already has the Object Selection tool: https://helpx.adobe.com/photoshop/using/tool-techniques/obje...
I tried Meta's photo segmentation tool and I didn't see anything magical in it.
I tried Meta's photo segmentation tool and I didn't see anything magical in it.
I haven't tried Meta's yet but Photoshop's tool though decent struggles with complex scenes
Very meta (pun intended) but is there a website that allows you to say "I want to find an AI engine / application that does X", where X might be any iteration of terms:
"magic wand tool that cuts an image from the background"
"like photoshop where you cut an object from the background"
"photo segmentation tool"
Google would be the de facto here, but its kind of worthless as its gamed by SEO.
"magic wand tool that cuts an image from the background"
"like photoshop where you cut an object from the background"
"photo segmentation tool"
Google would be the de facto here, but its kind of worthless as its gamed by SEO.
Are there any open source alternatives for this? Or still too early?
The model is open source
https://github.com/facebookresearch/segment-anything/
https://github.com/facebookresearch/segment-anything/
SAM itself _is_ open source: https://github.com/facebookresearch/segment-anything
ASL 2 license.
ASL 2 license.
I wrote a napari-plugin for it, https://github.com/JoOkuma/napari-segment-anything
You just need to have napari installed. I only tested on Linux, so it might not work on Windows/OSX.
You just need to have napari installed. I only tested on Linux, so it might not work on Windows/OSX.
This is the open source alternative.
give it 36 hours and we'll have segMenPoTader.cpp
Unrelated, but can we please get web browsers to enable users to totally turn off all auto play video and video encoding?
I hate websites like this where you deal with a minefield of video areas that might start playing audio at the smallest tap and drain your bandwidth for content you have zero desire to watch.
I hate websites like this where you deal with a minefield of video areas that might start playing audio at the smallest tap and drain your bandwidth for content you have zero desire to watch.
Firefox can block audio/video autoplay. You can set a default and have per-site exemptions: https://support.mozilla.org/en-US/kb/block-autoplay
Or news sites that play some annoying caster talking about it while you're trying to read and it follows you down the page. I'm here for an article not someone reading it to me.
I guess my ad blocker or some configuration protected me from it. I didn't encounter a single auto-play video or audio.
My firefox seems to not play it by default.
Im autistic and have the same Problem. I use chrome extensions which work fine