Warning: This is AI generated, probably a low end model as some of the content is outright nonsense eg:
"""
concept of MoE is quite prevalent (refer Outrageously Large Neural Networks: the Sparsely-Gated Mixture-of-Experts Layer), with Langchain’s high-level implementation of an LLMRouterChain, and notable low-level integrated examples
"""
No, that's not what I meant. I meant that in its reinforcement learning phase, GPT saw examples of "fix this text" style requests and was rewarded for doing a good job. That's different from seeing examples of typos and still predicting the right word which happens during the language model self supervised training. Both likely help it be good at it.
Yup totally plausible. Things like word (token) dropout and inserting random uniform noise into embeddings or just edit distance perturbations to the tokens are all well known but still Figure 1 looks extremely impressive.
GPT-4 was clearly trained to fix typos and handle not well written written requests. That much is visible directly from just using it within chatGPT UI in normal usage and fits common user scenarios (eg fix my bad draft). We know it was trained on social media data from Reddit much of which is not great writing either. Now I'm wondering if it was trained on (imperfectly) OCRed data too...
Transformer is the architecture. "Generative Pretrained" is just a term made up by the author to mean what everyone called for decades before and will call for decades after "Language Modelling". It was just a new way of saying "Language Modelling Transformer" that sounded cooler to the author and gave it cool initials. Coming up with cool names for models is hard.
These ToS, CSAM, moderation, etc are very orthogonal to the app itself. Why conflate them? Most non-hackernews people are completely unaware of them, so it won't be a factor in their choice of platform. Are we even sure other apps aren't doing similar things? Or have equally dangerous practices around password changes?
Yikes! good luck, hope it works out. This is scary. Gmail is still, by far, the best email app. Something like this could happen to me and, I suspect, a lot of other people.
I don't know what being in "tools for thought space" means but if it just mean "Roam or similar user", I can give my personal answer: I set up 2 or 3 templates when I started. The most used one I updated once since then. That's it. The time I spent doing this is essentially 0% of my Roam use time.
I can see how someone who checks out blogs/youtube about Roam-like tools might get the wrong impression but those are usually a business so they have to output content.
It also comes down to personality, I suspect some people have a tendency to optimize tooling as a form of procrastination.
Must've been quite the statistical masterpiece to disentangle it from the World Wars and the Great Depression. Off course extrapolating conclusions from early 20th century education and career progression to the 21st will be an even greater achievement.