Entrepreneur from Poland. Founded Szuku.pl, a polish people search engine, and EasyAsk - info-bot company. M.Sc at University of Warsaw. Currently involved in iPhone apps development - www.motivapps.com (first product - a Goal-Setting Workshop: www.motivapps.com/gsw )
That is not true - a model trained in the internet can both build verifiers to remove false/poor quality data from the next training, and build synthetic datasets that will supplement its training.
Similar to a human that wants to learn something and invents exercises to practice.
1-2 years ago it was a theory, but new models are trained, successfully, on synthetic datasets.
There isn’t a big difference between interpolation and extrapolation when the space has an immense amount of dimensions, and when you are free to modify the space at will.
This is a recurring sentiment but flawed, I think.
First of all, neural nets do nit return averages per se. They construct space between the points and extrapolate outside
of the points. So even if a point was not in their training data, they will be ok, in many situations, to acknowledge it.
Or in other words - LLMs don’t average. They construct world models. A novel thing that fits their world model will be accepted no prob. A thing that doesn’t may still be accepted but with
challenges.
The same is true though for humans, including scientists. There is a saying that science moves one grave at a time - because often prev gen of scientists needs to die off for a new idea to take root.
Or in yet other words - even if llms produced averages, an average
of a discontinuous set can lie outside of that set. And the set of all human ideas is very much discontinuous.
1M context - at least with most of the projects I ever worked with, 1M, or even 100k would be enough to explain in broad strokes the class/project/deployment structure, and a window of 200-500k to explain the specific issue at hand.
And seriously - what about people without hands? What about scammers pretending to be Google gaining access to my camera? What about blind people? What about people using the site in places where camera use is not allowed?
From my benchmarks, sadly, it doesn't seem to be the case much. Surprisingly. I found Sonnet comparable in speed to Opus (sic), but perhaps I was testing it wrong?
It’s not that difficult, it’s just a system prompt and a set of basic file edit/bash/etc tools.
Me, personally, I didn’t build it from scratch but I ported original CC from published sources into Python and extended it to match my own requirements.
It is most probably metabolism/genetics dependent. For me, I didn't have much withdrawals (aside from a first week), but even a month later I was insanely sluggish and it was difficult for me to function properly.
Just because it wasn’t hard for you doesn’t mean it isn’t hard for the others. Grom what you said it seems you just had some bad habits you had to fix and that’s it.
workIn-Progress.posterous.com