What's to say this isn't just a demonstration of memorization capabilities? For example, rephrasing the logic of the question or even just simple randomizing the order of the multiple choice answers to these questions often dramatically impacts performance. For example, every model in the Claude 3 family repeats the memorized solution to the lion, goat, wolf riddle regardless of how I modify the riddle.
Very cool. Are you familiar with plasmic.app? I'm expecting to integrate it into my next site build and would be curious for your take having built something in the same general category.
I'm attempting to write a function that splits a long document into shorter segments of text, splitting the text into the topics discussed as a step in a data processing pipeline prior to embedding the shorter segments of text for vector search.
I'm attempting to use v1.0 of the pomegranate python library as I get the impression it will be more performant than some of the other common options I looked at. Below is my code. I'm a self-taught developer just trying to solve a niche problem that's of interest to me so I've not used any of these libraries before or attempted to build a hidden markov model before so be gentle and many thanks for the help.