Show HN: Music Audio Search Engine Using OpenAI's Embeddings on GPT Descriptions(muzic-sage.vercel.app)
muzic-sage.vercel.app
Show HN: Music Audio Search Engine Using OpenAI's Embeddings on GPT Descriptions
https://muzic-sage.vercel.app/
43 comments
Hi, Freesound developer here, this is great! We're a small research team at a university[1], and we're also working on auto-captioning tasks that include trying to describe sound effects based on the audio content, although we're not doing anything with OpenAI or summarisation of metadata at the moment.
A reminder, we also have acoustic similarity using signal processing analysis and nearest neighbour lookup for all sounds[2]
Could you update the sound displays to include a link to the sound page on Freesound, for citation purposes? You can use https://freesound.org/s/[soundid] or use our embedded player. See the "Embed" option in the share box on the right -hand-side toolbar of any sound page.
Are you interested in integrating this into FS? Talk to us![3]
[1] https://www.upf.edu/web/mtg/
[2] https://freesound.org/people/lebaston100/sounds/243627/simil...
[3] https://github.com/mtg/freesound
A reminder, we also have acoustic similarity using signal processing analysis and nearest neighbour lookup for all sounds[2]
Could you update the sound displays to include a link to the sound page on Freesound, for citation purposes? You can use https://freesound.org/s/[soundid] or use our embedded player. See the "Embed" option in the share box on the right -hand-side toolbar of any sound page.
Are you interested in integrating this into FS? Talk to us![3]
[1] https://www.upf.edu/web/mtg/
[2] https://freesound.org/people/lebaston100/sounds/243627/simil...
[3] https://github.com/mtg/freesound
Thanks for sharing!
Some initial feedback:
1. Can you provide some example searches that you think or demonstrative of good matches?
2. On mobile safari, the search results overflow and the scrollbar flashes as the results are painted: https://i.postimg.cc/85RMrJxQ/81-BA5335-6-AD9-4277-93-CD-D50...
3. Add some links so people can reach out to you for feedback/bugs
Some initial feedback:
1. Can you provide some example searches that you think or demonstrative of good matches?
2. On mobile safari, the search results overflow and the scrollbar flashes as the results are painted: https://i.postimg.cc/85RMrJxQ/81-BA5335-6-AD9-4277-93-CD-D50...
3. Add some links so people can reach out to you for feedback/bugs
Thanks for the great feedback, I really appreciate it!
1. I'll definitely add some good examples, I think that will help with getting better sounds. 2. Working on improving the UI right now 3. Going to add this in the next push as well. For now reach out at [email protected]
1. I'll definitely add some good examples, I think that will help with getting better sounds. 2. Working on improving the UI right now 3. Going to add this in the next push as well. For now reach out at [email protected]
This is great! Some suggestions:
- Add download links with download attribute:
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download
- Make the grid responsive with CSS minmax (and media queries):
https://developer.mozilla.org/en-US/docs/Web/CSS/minmax
- Display a (visual?) score for closeness of match
- Add link to similar sounds (by the vector database closeness)
- Add query parameter so the results can be bookmarked
- Use HTML5 History API
- Show tags from freesound.org (and allow to filter by tags from search results)
- Generate visual previews for the sounds (like in Soundcloud)I don't know how to use it. I put in "boom boom boom" expecting thumping base sound and I got a completely unexpected beat structure instead. How the heck are you supposed to describe sounds? Is there a secret language?
This was fun to play with. I was initially expecting to be able to search for single notes or instrument sounds, e.g. I tried "warm synth tone single note C4" but afterwards I realised its library has mostly fixed tempo loops.
Like the idea, would be great if you linked back to freesound for each sample so I could explore the author's other sounds.
With my web audio hat on I'm imagining an interface that lets you mix, edit and add effects to the sounds you've found to help create new ones.
Thank you for sharing!
Like the idea, would be great if you linked back to freesound for each sample so I could explore the author's other sounds.
With my web audio hat on I'm imagining an interface that lets you mix, edit and add effects to the sounds you've found to help create new ones.
Thank you for sharing!
Thanks!
Editing the audio on the web would be really interesting. I'll look into this some more.
Editing the audio on the web would be really interesting. I'll look into this some more.
Update:
- Redoing UI
- Adding good prompt examples
- Planning on cleaning up code and open sourcing. If anyone is interested in contributing feel free to reach out at [email protected]
- Going to try to embed the audio files to create similarity clusters.
- Might add folder upload so that producers can upload large loop libraries and have their sounds clustered or allow them to search their library through natural language.
- If you have any ideas for cool things I could add or improve, feel free to leave it in a comment or reach out through my email!
- Redoing UI
- Adding good prompt examples
- Planning on cleaning up code and open sourcing. If anyone is interested in contributing feel free to reach out at [email protected]
- Going to try to embed the audio files to create similarity clusters.
- Might add folder upload so that producers can upload large loop libraries and have their sounds clustered or allow them to search their library through natural language.
- If you have any ideas for cool things I could add or improve, feel free to leave it in a comment or reach out through my email!
WOW, it gave me a great match on the first try. Here are some examples to test if you are not into music production:
"dark atmospheric pad"
"crispy sharp dark bass with a beat"
"groovy hi hat"
But!
"cowbells" I gotta have more cowbells!
"dark atmospheric pad"
"crispy sharp dark bass with a beat"
"groovy hi hat"
But!
"cowbells" I gotta have more cowbells!
I think you may need more than 3,000 grooves. I like the project idea.
I asked for "sad folorn trumpet sound over gentle spanish acoustic guitars"
I got: piano synth over drums weird beeping noises drums
I got: piano synth over drums weird beeping noises drums
Right now it only has grooves in the database. Going to add more sounds soon
We would love to help promote this? Ping me Joe dot Drumgoole at mongodb.com as we can get you some time on our podcast.
Nicely done.
On iOS mobile, the three per row causes overlap of the sound playback bars. (First per row's playback button is also off-screen, so cannot be used)
Easiest solution would likely just be one per row if the screen is small enough.
On iOS mobile, the three per row causes overlap of the sound playback bars. (First per row's playback button is also off-screen, so cannot be used)
Easiest solution would likely just be one per row if the screen is small enough.
Nice project and great to see more stuff built with the OP stack (OpenAI + Pinecone). I'm from Pinecone and if there's anything I can do to support, send me an email: [email protected]
Thanks Greg, Pinecone has been great!
Really like this project. Curious to learn more about the embedding process. Can you describe the process of generating the unique description a bit more?
I used the Freesound.org api (https://freesound.org/help/developers/) to download a bunch of sounds to MongoDB. Specifically, I selected to include the ID, Name, Tags, Description, Previews, Images, and Analysis in the response. I then looped through all these sounds and combined the response info into a prompt for GPT Davinci, instructing it to create a descriptive paragraph about the sound. I combined Davinci's description with the Freesound.org descriptions and embedded it using Ada. I inserted the embeddings into Pinecone with the audio preview url and ID as metadata. Then I just embed the search query and compare it with the sound embeddings (Pinecone allows you to select what type of similarity, I used cosine) and return the 25 most similar sounds.
Fascinating.
What if instead of reading a sound library, you had a music generator on the backend? What might an integration look like if it used the existing samples for description, as input for a "more like this" kind of feature
What if instead of reading a sound library, you had a music generator on the backend? What might an integration look like if it used the existing samples for description, as input for a "more like this" kind of feature
You could add visualization, thumbnails for sound. Because it's very hard to navigate.
This cannot be used without a government ID card in some countries:
- You require a Google account to use it.
- To create a Google account, the user needs a phone number.
- SIM cards are in some countries only allowed to be sold if you show your ID card. Landline phones require a bank account, and those also require an ID.
Now, call me old fashioned, but even considering that I have nothing to hide about my music taste, I am appalled at needing government approval to search for music :)
And yes, I know I am a party pooper here, but this pattern has already creeped all over all Internet services, so at which point in time should we start to point it out everywhere if not now?
Anyway, to be constructive: If you want me to be human I will happily solve dozens of captchas or have my CPU run hot for a while to generate hashcash :) No need to involve daddy Google.
- You require a Google account to use it.
- To create a Google account, the user needs a phone number.
- SIM cards are in some countries only allowed to be sold if you show your ID card. Landline phones require a bank account, and those also require an ID.
Now, call me old fashioned, but even considering that I have nothing to hide about my music taste, I am appalled at needing government approval to search for music :)
And yes, I know I am a party pooper here, but this pattern has already creeped all over all Internet services, so at which point in time should we start to point it out everywhere if not now?
Anyway, to be constructive: If you want me to be human I will happily solve dozens of captchas or have my CPU run hot for a while to generate hashcash :) No need to involve daddy Google.
Stop being overdramatic. You don't need a phone for a Google account. You can even use a fake name, birthday, gender, etc when setting up your account.
source: I have made several Google accounts, but have only tied my primary account to my phone.
source: I have made several Google accounts, but have only tied my primary account to my phone.
Depends on the region you're creating the account from. If it's a VPN, google definitely tries to get a phone number.
Then don't do that.
[deleted]
[deleted]
hndamien(2)
I used the Freesound.org API to upload over 3,000 grooves to MongoDB, and combined all the relevant data such as tags, title, description, BPM, etc. into OpenAI's Text-Davinci to generate a unique description of each sound. I then embedded these descriptions using the Ada Embeddings Model and inserted them into Pinecone DB vector database, making it easier to find the perfect sound for your next project.
This search engine is designed to save you time and make your music production process more efficient. Give it a try and see the difference it makes in your workflow. I would also appreciate any feedback on how I can improve the website and make it even more user-friendly.
EDIT: - Google Sign In removed - 25 total sounds now (only 5 before)