Yeah something like that. I am thinking of using audio to text converter using whisper apis, index the audio text in a vector DB and then perform string matching.
I have been working on building a site where you search by phrases which you have heard in some youtube shorts and are not able to remember which video was it, my project would give the users the videos in which the phrase has been said or mentioned