I retired a few years ago and now do a slew of things (photography, robotics, 3D printing, etc.) entirely for myself. I'll start a project, have a blast learning/building, then simply move on to the next thing.
Learning for the sake of learning is one of my favorite things in life.
> Who gets a sense of accoplishment from prompting an LLM?
I have a good friend who is a VP at a telecom company who has never written a line of code. He's been using Claude to create interactive web pages to help him understand parts of the company.
He was so excited when he got something to work he called me immediately.
I'm sure the code isn't what you or I would write, but it is good enough for my friend. That said, heaven help him if he loses access to Claude. ;-)
I went this route because I had difficulty visualizing the content of the Attention Is All You Need paper. After going through both books, I can now understand every part of that paper.
I'm currently working on a robotics project that uses Nvidia's GR00T N1 model, and I was able to understand the research paper. [0]
Well, as I suggested, working through the implementation yourself will give you that intuition. That said, I think the simplest way to explain why positional encodings are useful is that it gives the transformer just enough information to make attention meaningful without negatively impacting any parallel, content-based comparisons.
A vanilla self-attention layer is just a set of token vectors. Without positional info, swapping two identical embeddings changes very little about what attention can compute. We can "fix" this problem by using positional encodings. Text that has meaning isn't just a set of characters; the location and order of those characters is what provides meaning.
I have a BS in CS (and have been in the field for 25 years). I couldn't understand the transformer architecture until I built a few myself. Here are the books I worked through. I now feel I have a very good understanding of modern LLMs.
Not OP but I worked through Sebastian Raschka's "Build a Large Language Model (From Scratch)" [0] and Raj Abhijit Dandekar's "Build a DeepSeek Model (From Scratch)" [1] books.
I don't think there is anything in a transformer I couldn't explain in the smallest detail now.
I was an engineer at both MS and Apple, and wholeheartedly agree with you.
My question is, what happens to the people who use RTX cards for gaming? This new solution isn't meant for that. Do they need an "AI accelerator" and a gaming-centric GPU?
> "Our goal is to deliver unmetered intelligence to every home and every desk with Windows," said Satya Nadella, chairman and head of Microsoft.
Then:
> However, Ian Fogg, Research Director at industry analyst firm FDM CCS Insight said the change was "likely to come with a significant price tag" and Nvidia would be targeting "those looking for workstation-class performance".
As someone who's been building developer tools (Visual Studio and Xcode) for 25 years, I don't have a perspective problem. We were doing "code completion" back in the 90s and could never have predicted that an LLM would write code at the current level of quality.
My point is that with every new model release, the expectations grow. I don't know how else to say that.
He didn't even know that there was a solution to the performance issues. He simply assumed that processing data took that long.
I think it is great that he now has this capability, but a total ignorance of software engineering is going to continually bite this type of user. Instead of questioning Claude's solution, my friend thought he just needed a faster computer.
He was also using very sketchy Python imports when much safer, more mature options are available. Not knowing that you shouldn't use just any random Python package is a ticking time bomb... especially when his machine is connected directly to his corporate intranet.
I was fortunate to get to spend time with woz when I worked at Apple. He's the type of person who is practically silent during a meeting. Then, towards the end, he spoke up and would literally solve the problem we'd been struggling with the entire time.
He's one of the nicest, most down-to-earth people I've ever worked with.
I have a friend who is VP at a major telecom company. He has no technical experience but has been using Claude to create data analysis apps. He was complaining that it took three hours to process certain datasets, so I took a look.
He had Claude essentially create a 300MB json file and was doing all of the data processing on that data directly.
It never occurred to him, or Claude, that there were other ways to operate on that data. It took me less than 10 minutes to get that processing time down to under a minute.
These are the type of issues that worry me about vibe coding.
> Which statement of mine do you think is not settled law?
I object to your statement that "copyright doesn’t get involved at all" when that is objectively untrue. If that was true, many of the world's largest companies wouldn't be spending tens of millions of dollars to have that question answered in court. Go to any law-focused forum, and you will find attorneys arguing over these questions.
To train a model using a book, you must first obtain a copy of that book. Did OpenAI purchase a copy of every book not already in the public domain used during training? They did not.
Some of the suits I mentioned claim that OpenAI literally stole copies of books to train its models.
My point is that the copyright question has not been answered. If the NYT, et. al. win, it will be a watershed moment for how AI companies pay for training data moving forward.
To do that training, you must first obtain the item with the content you require. Did OpenAI purchase a copy of every book they trained their models on?
Answer: They did not. That is literally why there are dozens of ongoing lawsuits in progress.
I appreciate your comment, but you answered as if this question had been answered legally. It has not.
The New York Times is suing both OpenAI and Microsoft for copyright infringement. The Authors Guild is suing OpenAI. Getty Images is suing Stability AI. Disney is suing Midjourney. Universal Music Group and Sony have filed suits against multiple AI companies.
> so copyright doesn’t get involved at all.
The dozens of ongoing cases that discredit that statement.
I'd like to understand why I can't use a song in one of my videos without permission/payment, but an AI company can train models using that song without having either.
I'm not anti-AI. I'd just like to see companies play by the rules everyone else has to follow.
I understand where you are coming from, but at least when I was there, we were still trying to develop solutions that had never been implemented at that scale before (just like Anthropic today). I helped create the first version of Visual Studio (Boston). People tend to forget that even by the 90s we still didn't really understand how to solve a lot of the main technical problems. That's what I loved about the work. Everything seems easy/obvious after the fact.
When I left MS, a full Windows build was about 18M LOC. The fact that 18 million lines of code, written by tens of thousands of engineers, worked at all was a mini miracle.
With regard to compensation: like Karpathy, I had already earned enough to be comfortable for the rest of my life. Once money stopped being the primary driver, I was able to focus on what made me happy. Building things, even if you don't like them, brought me happiness and fulfillment. I hope Andrej finds the same at Anthropic.
Learning for the sake of learning is one of my favorite things in life.