What's to say this isn't just a demonstration of memorization capabilities? For example, rephrasing the logic of the question or even just simple randomizing the order of the multiple choice answers to these questions often dramatically impacts performance. For example, every model in the Claude 3 family repeats the memorized solution to the lion, goat, wolf riddle regardless of how I modify the riddle.
Very cool. Are you familiar with plasmic.app? I'm expecting to integrate it into my next site build and would be curious for your take having built something in the same general category.
I'm attempting to write a function that splits a long document into shorter segments of text, splitting the text into the topics discussed as a step in a data processing pipeline prior to embedding the shorter segments of text for vector search.
I'm attempting to use v1.0 of the pomegranate python library as I get the impression it will be more performant than some of the other common options I looked at. Below is my code. I'm a self-taught developer just trying to solve a niche problem that's of interest to me so I've not used any of these libraries before or attempted to build a hidden markov model before so be gentle and many thanks for the help.
Depends a bit what your goals are. But I think generalist are undervalued on average. Range: Why Generalists Triumph in a Specialized World by David Epstein is all about this and he says it better than I can so it’s easiest to just point you there.
There seem to be a few "long rang transformer options" (https://huggingface.co/blog/long-range-transformers). They're slower (not an issue for my use case though). I've heard one talk on this issue mention that there are some "tricks" you can use with web docs (html) like hrefs, to get creative. But more commonly from what I've read, people try different approaches for segmenting the documents, then embed those segments. Heard good things about using a Hidden Markov Model for the segmenting approach.
I hadn't seen that approach to evaluating search engines. But looking at the github repo I'm also not quite following how I would use this/what the standard approach is for scoring relevance ranking approaches, and how this approach differs from that standard approach. If it's not too much trouble a tldr on that would be a really useful intro.
I do intend to have the UX be setup in a way for the user base to sort of re-rank or at least provide some feedback on which results were irrelevant to help with re-training the model over time. For certain applications where very limited domain knowledge is required (for example, is this a hardware product or a sofware product? - or is this a product or a service?), I can also use mechanical turk or similar to label data and I fully intend to do that.
I'm a minimally technical entrepreneur that could use some guidance on piecing together an MVP for a search product.
Some quick background on the idea (also in the linked document):
What Users Want: Ability to quickly generate a list of company websites that match their custom industry definition/query.
MVP Goal: Complex enough to properly validate/reject the idea, simple enough to not be prohibitively expensive or too many man hours to reach validation/rejection step. Ideally, I can add features/complexity to the implementation post validation without completely changing the architecture (for example, adding vector search post initial validation with something like BM25)
Example user input queries:
software for catering businesses
crane inspection service
laboratory reagent suppliers
Output for each case would be a list of relevant businesses they can further filter by relevant criteria like employee count, location, etc.
Some key questions I could use some help answering are:
a) At present, how much value will vector search add beyond BM25/BM25F or similar?
b) Given the recent rate of progress in LLMs, I'm expecting embeddings for search to improve at a similar rate, and therefore assuming I should expect to be implementing vector based search in the near future even if it's not part of the MVP.
I've share some of my research so far in the linked document. Would really appreciate some feedback on it. How would you build this MVP if you were trying to do it bootstrapped/solo?