Would love to set up everything on Kamal but just feels like slightly too much management cost in maintaining a database yourself for mission critical stuff. If I'm going to use a managed DB, might as well do the same for the container?
Maybe if I just ran with sqlite and a preconfigured litestream inside the main container that'd simplify things a fair bit, but then you're immediately locked onto a single machine.
This is silly. Look at the loss and benchmark curves for the Pythia suite of models - the smaller models certainly did saturate and in fact began worsening.
2T not saturating on a 7B is very different from 3T on a 1B.
I suspect the only solid solution is OpenAI themselves storing all text their models generates, and providing an API which will return whether they've outputted a specific string (or a similar one) in the past.
A lot of suggestions here talk about the consistent stylistic choices that ChatGPT makes, like it's lists or other particular mannerisms. I'd argue these are simply artefacts of it being fine-tuned on a large number of 'well-behaved' examples from Open AI. This phenomena is called partial mode collapse, this article does a great job discussing it with respect to GPT-3 [0].
Of course you could train a model to detect when this mode-collapse occurs to detect ChatGPT. The un-finetuned model, however, does not have these problems, so it's only a matter of OpenAI improving their fine-tuning dataset to return to an 'undetectable' AI.
My naive belief is that VC's invest more in companies that they think will succeed (i.e. larger funding rounds).
Is it possible the model is partially latching onto this, meaning their result could actually be just saying "companies that VC's back more are also the one's that succeed", presumably because VC's have some of their own reasonable criteria for doing this.
I didn't see this discussed in the paper, I wonder if it's results would be as strong if they excluded size of funding rounds as an input.
Maybe if I just ran with sqlite and a preconfigured litestream inside the main container that'd simplify things a fair bit, but then you're immediately locked onto a single machine.