AI Engineer Reading List(latent.space)
latent.space
AI Engineer Reading List
https://www.latent.space/p/2025-papers
70 comments
AI engineering is more applied than research and in that regard this list is great.
swyx and teams podcast, newsletter and discord has been the highest signal to noise ratio for keeping up and learning.
swyx and teams podcast, newsletter and discord has been the highest signal to noise ratio for keeping up and learning.
thank you! very kind. much to improve.
By the way, I think this is a fantastic reading list for creating AI products, and especially for staying updated on the latest in the AI space. However, it feels a bit scattered and might be hard for beginners to follow, IMO.
I read your book, The Coding Career Handbook, we need something similar for AI Engineering! I really enjoyed it. Thank you for creating and sharing such high-quality multimodal content :)
I read your book, The Coding Career Handbook, we need something similar for AI Engineering! I really enjoyed it. Thank you for creating and sharing such high-quality multimodal content :)
wow thanks for reading it! yeah i think i should just write the new book for ai engineering i guess but Chip Huyen already took the oreilly slot lol
Of the resources you mention, #2 is probably the best starting point for someone who wants to start building software soon. Karpathy's videos fast.ai's courses may also fit that purpose.
But the other books (#1, #3, #4) seem like they're intended for those who want to understand all the math. Many people don't want (or need) a full understanding of how all this works. They can provide significant value to their employers with some knowledge of how machine learning works (e.g. the basics of CNNs and RNNs), and some intuitions/vibes about SOTA LLMs, even if they don't understand transformers or other modern innovations.
But the other books (#1, #3, #4) seem like they're intended for those who want to understand all the math. Many people don't want (or need) a full understanding of how all this works. They can provide significant value to their employers with some knowledge of how machine learning works (e.g. the basics of CNNs and RNNs), and some intuitions/vibes about SOTA LLMs, even if they don't understand transformers or other modern innovations.
Is there a textbook like [1] or [4], which also incorporates PyTorch into learning?
Dive into Deep Learning is implemented using various libraries such as PyTorch, NumPy/MXNet, JAX, and TensorFlow.
Here’s an example: https://d2l.ai/chapter_natural-language-processing-pretraini...
Here’s an example: https://d2l.ai/chapter_natural-language-processing-pretraini...
This resource is not as profound compared to the other 2, unfortunately.
I thought the same too...
I read Deep Learning by Goodfellow and Deep Learning with TensorFlow 2 and Keras for practical stuff. I am still thinking if I should do the D2L for additional practice in my free time, though.
I read Deep Learning by Goodfellow and Deep Learning with TensorFlow 2 and Keras for practical stuff. I am still thinking if I should do the D2L for additional practice in my free time, though.
I don't know what an "AI Engineer" is, but, is reading research papers actually necessary if the half life of the relevancy of many of these papers is only a few months until the next breakthrough happens?
I have a feeling, unless you're dabbling at the cutting edge of AI, there's no point in reading research papers. Just get a feel for how these LLMs respond then build a pretty and user friendly app on top of them. Knowing the difference between "multi head attention" and "single head attention" isn't very useful if you're just using OpenAI or Groq's API.
Am I missing something here? I'd love to know where I'm wrong
I have a feeling, unless you're dabbling at the cutting edge of AI, there's no point in reading research papers. Just get a feel for how these LLMs respond then build a pretty and user friendly app on top of them. Knowing the difference between "multi head attention" and "single head attention" isn't very useful if you're just using OpenAI or Groq's API.
Am I missing something here? I'd love to know where I'm wrong
> I don't know what an "AI Engineer" is, but, is reading research papers actually necessary
Let's put it this way: if even half the people who call themselves "AI Engineers" would read the research in the field, we'd have a lot less hype and a lot more success in finding the actual useful applications of this technology. As is, most "AI Engineers" assume the same thing you do and consider "AI Engineering" to be "I know how to plug this black box into this other black box and return the result as JSON! Pay me!". Meanwhile most AI startups are doomed from the start because what they set out to do is known to be a bad fit.
Let's put it this way: if even half the people who call themselves "AI Engineers" would read the research in the field, we'd have a lot less hype and a lot more success in finding the actual useful applications of this technology. As is, most "AI Engineers" assume the same thing you do and consider "AI Engineering" to be "I know how to plug this black box into this other black box and return the result as JSON! Pay me!". Meanwhile most AI startups are doomed from the start because what they set out to do is known to be a bad fit.
> I know how to plug this black box into this other black box and return the result as JSON!
To be fair, most of software engineering is this.
To be fair, most of software engineering is this.
Tbf- most of [any] engineering is like this.
But most 'engineering' is not engineering.
Care to explain your perspective? "Engineering" can be a bit of a fuzzy definition. To some it means "building something". To others, it requires the application (and understanding!) of mathematic and scientific principles to build something.
I would disagree that most engineering is not involved in building something...whether most engineers understand the math/science behind it is debatable.
I would disagree that most engineering is not involved in building something...whether most engineers understand the math/science behind it is debatable.
Okay, now take a slightly imbalanced stance: What is most software engineering?
I don't know, but if I say it's about working with things you don't fully understand people seem to trust me.
I kind of see this the opposite way...
Or rather, I guess I feel like it's a sign of the immaturity of the space that it is still kind of unclear (at least it is to me) how to build useful things without reading all the research papers.
To me, it seems like there is an uncanny valley between "people who are up on all the papers in this reading list" and "people who are just getting a feel for how these LLMs respond and slapping a UI on top".
Maybe it kind of reminds me of the CGI period of the web. The "research papers" side is maybe akin to all the people working on networking protocols and servers necessary to run the web, and the "slap a UI over the llm APIs" is akin to those of us slinging html and perl scripts.
You could make ok stuff that way, without needing to understand anything about TCP. But it still took a little while for a more professionalized layer to mature between those two extremes.
I feel like maybe generative AI is in the early days of that middle layer developing?
Or rather, I guess I feel like it's a sign of the immaturity of the space that it is still kind of unclear (at least it is to me) how to build useful things without reading all the research papers.
To me, it seems like there is an uncanny valley between "people who are up on all the papers in this reading list" and "people who are just getting a feel for how these LLMs respond and slapping a UI on top".
Maybe it kind of reminds me of the CGI period of the web. The "research papers" side is maybe akin to all the people working on networking protocols and servers necessary to run the web, and the "slap a UI over the llm APIs" is akin to those of us slinging html and perl scripts.
You could make ok stuff that way, without needing to understand anything about TCP. But it still took a little while for a more professionalized layer to mature between those two extremes.
I feel like maybe generative AI is in the early days of that middle layer developing?
Even before the amazing achievement of LLM, there were millions of "ML engineers" on LinkedIn, per some stats about LinkedIn jobs. I'll bet a single digit percent of them could even derive the math of linear regression or every implemented a single ML algorithm from scratch. Not that it is wrong, mind you, but it means it's unlikely for half the "AI engineers" to read research papers.
A lot of people did a MOOC for the CV points.
> if even half the people who call themselves "AI Engineers" would read the research in the field, we'd have a lot less hype and a lot more success in finding the actual useful applications of this technology
As someone working in the area for a few years now (both on the product and research side), I strongly disagree. A shocking number of papers in this area are just flat out wrong. Universities/Research teams are churning out garbage with catchy titles at such a tremendous rate that reading all of these papers will likely leave one understanding less than if they read none.
The papers in this list are decent, but I wouldn't be shocked if the conclusions of a good number of them were ultimately either radically altered or outright inverted as we learn more about what's actually happening in LLMs.
The best AI engineers I've worked with are just out there experimenting and building stuff. A good AI engineer definitely has to be working closely to the model, if you're just calling an API you're not really an "AI Engineer" in my book. While most good AI engineers have likely accidentally read most of these paper through the course of their day job, they tend to be reading them with skepticism.
A great demonstration of this is the Stable Diffusion community. Hardly any of the innovation in that space is even properly documented (this, of course, is not ideal), much less used for flag planting on arXiv. But nonetheless the generative image AI scene is exploding in creativity, novel applications, and shocking improvements all with far less engineering/research resources devoted to the task than their peers in the LLM world.
As someone working in the area for a few years now (both on the product and research side), I strongly disagree. A shocking number of papers in this area are just flat out wrong. Universities/Research teams are churning out garbage with catchy titles at such a tremendous rate that reading all of these papers will likely leave one understanding less than if they read none.
The papers in this list are decent, but I wouldn't be shocked if the conclusions of a good number of them were ultimately either radically altered or outright inverted as we learn more about what's actually happening in LLMs.
The best AI engineers I've worked with are just out there experimenting and building stuff. A good AI engineer definitely has to be working closely to the model, if you're just calling an API you're not really an "AI Engineer" in my book. While most good AI engineers have likely accidentally read most of these paper through the course of their day job, they tend to be reading them with skepticism.
A great demonstration of this is the Stable Diffusion community. Hardly any of the innovation in that space is even properly documented (this, of course, is not ideal), much less used for flag planting on arXiv. But nonetheless the generative image AI scene is exploding in creativity, novel applications, and shocking improvements all with far less engineering/research resources devoted to the task than their peers in the LLM world.
Couldn't agree more with you. At the end of the day the people building the most successful products, are too be busy to be formalizing their experiments into research papers. While I have respect for academic researchers, I think their perspective is fundamentally very limited when it comes to AI engineering. The space is just too frothy.
Careful now. Don't want to upset the future generation of unemployed prompt engineers.
Investors can't tell the difference between hype and (future) results.
hi! author here.
> Just get a feel for how these LLMs respond then build a pretty and user friendly app on top of them.
as you know, "just" is a very loaded word in software engineering. The entire thesis of AI Eng is that this attidude of "just slap a UI on an LLM bro whats so hard" is a rapidly deepening field, with its own stack and specialization (which, yes, some if not much of which is unnecessary, vc funded hypey complexity merchantism, but some of which is also valid), and if you do not take it seriously, others will, and do so running rings around those who have decided to not even try to push this frontier, passively waiting for model progress to solve everything.
i've seen this play out before in underappreciated subfields of engineering that became their own thing, with their own language, standard stack, influencers, debates, controversies, IPOs, whole 9 yards.... frontend eng, mobile eng, SRE, data eng, you name it. you just have to see the level and quality of work that these people are doing that is sufficiently distinct from MLE and product/fullstack webdev to appreciate that it probably deserves its own field of study, and while it will NEVER be as prestigious as AI research, there will be a ton more people employed in these roles than there can be in research and thats a perfectly fine occupation too.
I'm even helping instruct a course about it this week as it happens if you want to see what a practical syllabus for it looks like https://maven.com/noah-hein/ai-engineering-intro
> Just get a feel for how these LLMs respond then build a pretty and user friendly app on top of them.
as you know, "just" is a very loaded word in software engineering. The entire thesis of AI Eng is that this attidude of "just slap a UI on an LLM bro whats so hard" is a rapidly deepening field, with its own stack and specialization (which, yes, some if not much of which is unnecessary, vc funded hypey complexity merchantism, but some of which is also valid), and if you do not take it seriously, others will, and do so running rings around those who have decided to not even try to push this frontier, passively waiting for model progress to solve everything.
i've seen this play out before in underappreciated subfields of engineering that became their own thing, with their own language, standard stack, influencers, debates, controversies, IPOs, whole 9 yards.... frontend eng, mobile eng, SRE, data eng, you name it. you just have to see the level and quality of work that these people are doing that is sufficiently distinct from MLE and product/fullstack webdev to appreciate that it probably deserves its own field of study, and while it will NEVER be as prestigious as AI research, there will be a ton more people employed in these roles than there can be in research and thats a perfectly fine occupation too.
I'm even helping instruct a course about it this week as it happens if you want to see what a practical syllabus for it looks like https://maven.com/noah-hein/ai-engineering-intro
[deleted]
There is multiple levels of AI engineers:
1. The actual deep ML researchers that work on models 2. The "AI engineer" who creates products based on LLM's 3. The "AI researchers" who basically just stack LLM's together and call it something like Meta-Cognitive Chain-of-Thought Advanced Reasoning Inteligence or whatever it is.
1. The actual deep ML researchers that work on models 2. The "AI engineer" who creates products based on LLM's 3. The "AI researchers" who basically just stack LLM's together and call it something like Meta-Cognitive Chain-of-Thought Advanced Reasoning Inteligence or whatever it is.
> 1. The actual deep ML researchers that work on models
> 3. The "AI researchers" who basically just stack LLM's together and call it something like Meta-Cognitive Chain-of-Thought Advanced Reasoning Inteligence or whatever it is.
I actually think that working purely within the traditional neural nets model is starting to hit against its limits and the most fruitful directions for research are systems that incorporate and modify LLMs on-line, among other systems, despite your unserious characterization of this class of research.
> 3. The "AI researchers" who basically just stack LLM's together and call it something like Meta-Cognitive Chain-of-Thought Advanced Reasoning Inteligence or whatever it is.
I actually think that working purely within the traditional neural nets model is starting to hit against its limits and the most fruitful directions for research are systems that incorporate and modify LLMs on-line, among other systems, despite your unserious characterization of this class of research.
Why are there two research branches?
Seems like there's one AI engineer, which is b. The other two are researchers, one doesn't even focus on AI since ML covers a broader swath of disciplines.
Seems like there's one AI engineer, which is b. The other two are researchers, one doesn't even focus on AI since ML covers a broader swath of disciplines.
I consider myself to be an (occasional) user of AI services like the ones OpenAI and others provide. I've learned how to consume the services reasonably effectively, and make good use of them, but that's about it. I am not an AI engineer.
Similarly I know how to call cryptography libraries to get my passwords hashed using a suitable cipher before storing them. I don't understand the deep math behind why a certain cipher is secure, but that's fine. I can still make good use of cryptographic functions. I'm not a cryptography engineer either :).
My take on it is that if you should call yourself any kind of "XYZ Engineer", you should be able to understand the inner workings of XYZ.
This reading list is most likely (mostly) for those who want to get a really deep understanding and eventuellt work on contributing to the "foundational systems" (for a lack of a better word) one day.
Hope that helps.
Similarly I know how to call cryptography libraries to get my passwords hashed using a suitable cipher before storing them. I don't understand the deep math behind why a certain cipher is secure, but that's fine. I can still make good use of cryptographic functions. I'm not a cryptography engineer either :).
My take on it is that if you should call yourself any kind of "XYZ Engineer", you should be able to understand the inner workings of XYZ.
This reading list is most likely (mostly) for those who want to get a really deep understanding and eventuellt work on contributing to the "foundational systems" (for a lack of a better word) one day.
Hope that helps.
i mostly agree w you, but theres a wide spectrum of “understand the inner workings” given rising complexity.
consider:
- does a React/frotnend engineer need to know everything about react internals to be good at their job?
- does a commercial airline pilot need to know every single subsystem in order to do their job?
- do you, a sophisticated hackernewsian, really know how your computer works?
more knowledge is always (usually) better but as a thing diffuses into practice and industry theres a natural stopping point that “technician” level people reach that is still valuable to society bc of relative talent supply and demand.
consider:
- does a React/frotnend engineer need to know everything about react internals to be good at their job?
- does a commercial airline pilot need to know every single subsystem in order to do their job?
- do you, a sophisticated hackernewsian, really know how your computer works?
more knowledge is always (usually) better but as a thing diffuses into practice and industry theres a natural stopping point that “technician” level people reach that is still valuable to society bc of relative talent supply and demand.
> - does a React/frotnend engineer need to know everything about react internals to be good at their job?
Yes? Well, not everything (which I define as being able to implement React from scratch). But if you want to do good work, and be able to fix those pesky bugs which result from the arcane behavior of the framework itself, then you better know your stuff.
Besides, in practice very few people understand the most basic stuff about React. Just recently I had to explain to a veteran frontend dev what list virtualization was and why it's not a good idea to display a list of 100k items directly.
Yes? Well, not everything (which I define as being able to implement React from scratch). But if you want to do good work, and be able to fix those pesky bugs which result from the arcane behavior of the framework itself, then you better know your stuff.
Besides, in practice very few people understand the most basic stuff about React. Just recently I had to explain to a veteran frontend dev what list virtualization was and why it's not a good idea to display a list of 100k items directly.
I personally found that people need to understand the layer of the stack they're working on (e.g. a frontend dev should understand React). Going a layer higher or lower (or two) seems only to be handy for troubleshooting, debugging or you're simply having an expanded role.
> does a commercial airline pilot need to know every single subsystem in order to do their job?
Not a great comparison. First off, nobody is suggesting that a self-purported "AI Engineer" has to understand EVERY SINGLE SUBSYSTEM, but they should still have a strong command of the internal workings of the modern foundational material (transformers, neural networks, latent space, etc.) to style themselves as such.
The better question is "does an aviation mechanic need to understand the internal systems of an airplane?" and the answer is a resounding yes.
Not a great comparison. First off, nobody is suggesting that a self-purported "AI Engineer" has to understand EVERY SINGLE SUBSYSTEM, but they should still have a strong command of the internal workings of the modern foundational material (transformers, neural networks, latent space, etc.) to style themselves as such.
The better question is "does an aviation mechanic need to understand the internal systems of an airplane?" and the answer is a resounding yes.
[deleted]
>- does a commercial airline pilot need to know every single subsystem in order to do their job?
Haha explain this one to the APDs (aircrew program designee, the people signing off training at airlines) please.
Every airline pilot has their horror stories of being asked how many holes are in the alternate static port of some aircraft they've flown. Or through bolts on the wheel hub, or how many plys of glass on the side cockpit window, or the formula for calculating hydroplane speed, or the formula for calculating straight line distance to the horizon from altitude of X... it just goes on endlessly.
I do agree with your post overall though.
Haha explain this one to the APDs (aircrew program designee, the people signing off training at airlines) please.
Every airline pilot has their horror stories of being asked how many holes are in the alternate static port of some aircraft they've flown. Or through bolts on the wheel hub, or how many plys of glass on the side cockpit window, or the formula for calculating hydroplane speed, or the formula for calculating straight line distance to the horizon from altitude of X... it just goes on endlessly.
I do agree with your post overall though.
We covered that too: https://www.latent.space/p/ai-engineer
A nice overview. In short, I would describe an AI Engineer as someone who integrates existing AI tools and libraries into a reliable system that is fielded in a production environment. They also must know how to tune AI components and assess them for performance, decline/drift, and failure. Most AEs have a MS or less (CS, data science, statistics, etc), since it's not really a research role. Finally, AI Engineers don't invent AI tools or fix what's missing/broken within them. That's the role of an AI scientist.
I think you need to have a pretty good understanding of the underlying workings of at least a couple of modern models. You should also follow along closely enough to see if major changes happen in the way they are built.
But watching every new paper? Nah, that's mostly only useful if you have a large enough amount of compute to try them out. And most of us don't have that anyway.
But watching every new paper? Nah, that's mostly only useful if you have a large enough amount of compute to try them out. And most of us don't have that anyway.
The terms are being diluted hard. Everyone from a plumber with a no-code RAG automation tool to a FAANG phd can and have claimed the role “AI engineer”, and it can be true depending on the context.
The new “Data Scientist”!
I think AI blurs it more than ever, as it becomes easier to use. A chat makes any rando a superhero in the right context, if others don’t know it.
Eg a 90 year old in a care home. You learn the ins and outs of any service, you’re the local expert. You don’t even need Excel, just a phone. Very many people who have never heard of deep learning have built chatbots for small retail shops. Drag in a few FAQ docs to the context store and click “go”.
Eg a 90 year old in a care home. You learn the ins and outs of any service, you’re the local expert. You don’t even need Excel, just a phone. Very many people who have never heard of deep learning have built chatbots for small retail shops. Drag in a few FAQ docs to the context store and click “go”.
There’s research papers on system design involving llms that would definitely be useful in practical contexts.
An AI Engineer is someone who builds AI products, just as a Software Engineer is someone who builds software products.
Hope that helps!
Hope that helps!
all those PM's and middle layers are going to be an "AI Engineer" soon, so now you know
You're not necessarily wrong but maybe a bit naive about what cutting edge research entails.
At least for 60% of those papers you can't reproduce the results....
Know the terminology so you can discern the flow... knowing the flow wil helo you orchestrate agentic compute, to anticipative memory.. for precursor compute
hi! author here! putting together a list like this is intimidating - for every thing i pick there are a dozen other suitable candidates so please view this as a curriculum with broadly prescriptive weightings, with the understanding that the CURRENTLY_RELEVANT_PAPER is always a moving pointer rather than fixed reference.
we went thru this specific reading list in our paper club: https://www.youtube.com/watch?v=hnIMY9pLPdg
if you are interested in a narrative version.
we went thru this specific reading list in our paper club: https://www.youtube.com/watch?v=hnIMY9pLPdg
if you are interested in a narrative version.
This is a great list - I was wondering if there was a less research oriented, more experimental or practical reading list that you're planning as well - some things off the top of my head:
- Actual examples of Fine tuning of LLMs or making merges - usually talked about in r/localLlama for specific use cases like role playing or other scenarios that instruction tuned LLMs are not good at. Jupyter notebook or blog post would be great here.
- Specifically around Agents & Code generation - Anthropic's post about SWE-bench verified gives a very practical look at writing a coding agent https://www.anthropic.com/research/swe-bench-sonnet with prompts, tool schema and metrics.
- The wide amount of Loras and fine tunes available on civitai for image models - a guide on making a custom one that you can use in ComfyUI.
- State of the art in audio models in production - Elevenlabs seems to still be the best for closed platforms, but there are some options for open source voice cloning, TTS, or even text to speech with very small parameter models (kokoro 82M).
- Actual examples of Fine tuning of LLMs or making merges - usually talked about in r/localLlama for specific use cases like role playing or other scenarios that instruction tuned LLMs are not good at. Jupyter notebook or blog post would be great here.
- Specifically around Agents & Code generation - Anthropic's post about SWE-bench verified gives a very practical look at writing a coding agent https://www.anthropic.com/research/swe-bench-sonnet with prompts, tool schema and metrics.
- The wide amount of Loras and fine tunes available on civitai for image models - a guide on making a custom one that you can use in ComfyUI.
- State of the art in audio models in production - Elevenlabs seems to still be the best for closed platforms, but there are some options for open source voice cloning, TTS, or even text to speech with very small parameter models (kokoro 82M).
welcome to write a guest post around it!
Great work!
I am out of my depth when it comes to reading papers, but I second 'The Prompt Report' from your list.
It gives a great taxonomy that helped me understand the space of prompting techniques better.
I am out of my depth when it comes to reading papers, but I second 'The Prompt Report' from your list.
It gives a great taxonomy that helped me understand the space of prompting techniques better.
he was great on the pod too https://www.latent.space/p/learn-prompting
Hey Swyx, this list of papers is really interesting - I like that it's an opinionated list. Why isn't there more about model implementation though? I'd have expected engineers in this field want to know pytorch, cuda, JAX, maybe even XLA?
because of the growth of the field and proliferation of closed model apis, that is out of scope for AI Engineer now - save that for the MLE and RSes
I appreciate the effort put into curating and maintaining this list, good job on that!
I’m curious, is there also some specific existing “AI Researcher Reading List” you would personally recommend? Or do you plan on making and maintaining one?
I’m curious, is there also some specific existing “AI Researcher Reading List” you would personally recommend? Or do you plan on making and maintaining one?
im no researcher so no, but i would start with the llm courses at stanford, uc berkeley, stanford, and princeton.
Heh, always fascinating to see how the term “AI” has been swallowed nigh-completely by the recent exciting developments in DL. All those papers and not a single mention of Russell & Norvig, Minsky, Shannon, Lenat, etc.!
I’m sure it’s a great list for what it is, I just wanted to be pedantic for a bit ;). If you’re interested in an introduction to AI as a broader topic, most graduate courses use the same book (Russel & Norvig) and others may publish their syllabi online.
I’m sure it’s a great list for what it is, I just wanted to be pedantic for a bit ;). If you’re interested in an introduction to AI as a broader topic, most graduate courses use the same book (Russel & Norvig) and others may publish their syllabi online.
i’ll go even further, i dont include attention is all you need in my list. i think theres always a necessary intellectual history, but sometimes when you just want to do a specific job, you just want the current thing, and you can backfill the older stuff later (which yes can bite you in the ass if you are ignorant)
From the article:
> 1. GPT1, GPT2, GPT3, Codex, InstructGPT, GPT4 papers. Self explanatory. (...)
> 2. Claude 3 and Gemini 1 papers to understand the competition. (...)
> 3. LLaMA 1, Llama 2, Llama 3 papers to understand the leading open models. (...)
I agree that you should have read most of these papers at the time, when they were released, but I wonder if it would be that useful to read them now.
Perhaps it would be better to highlight one or two important papers from this section?
> 1. GPT1, GPT2, GPT3, Codex, InstructGPT, GPT4 papers. Self explanatory. (...)
> 2. Claude 3 and Gemini 1 papers to understand the competition. (...)
> 3. LLaMA 1, Llama 2, Llama 3 papers to understand the leading open models. (...)
I agree that you should have read most of these papers at the time, when they were released, but I wonder if it would be that useful to read them now.
Perhaps it would be better to highlight one or two important papers from this section?
I think most of the instruction fine-tuning methods for oss models stem from Alpaca, so it should be included: https://crfm.stanford.edu/2023/03/13/alpaca.html
And the one referenced in there on synthetic data generation: https://arxiv.org/abs/2212.10560
And the one referenced in there on synthetic data generation: https://arxiv.org/abs/2212.10560
This is a great survey. Combine it with the courses below for best results:
https://www.trybackprop.com/blog/top_ml_learning_resources
https://www.trybackprop.com/blog/top_ml_learning_resources
If you are not on that hype-train, yet, then
don't waste time skimming over, reading and understanding any LLM and AI papers.
Read about ELIZA. Build your own.
Get Tensors, Vectors, Fields, Linguistics, Computer Architectures, Networks.
Focus on the subjects themselves, not them in the context of Neural Networks, "Deep Learning" et al.
don't waste time skimming over, reading and understanding any LLM and AI papers.
Read about ELIZA. Build your own.
Get Tensors, Vectors, Fields, Linguistics, Computer Architectures, Networks.
Focus on the subjects themselves, not them in the context of Neural Networks, "Deep Learning" et al.
The list seems deliberately focussed on learning practical topics and not the foundations. I think that's what makes it interesting - there are any number of places recommending you start by learning linear algebra, stats, probability...
Personally I like to learn the foundations but there's genuinely room for useful knowledge of SOTA techniques even without the foundations. To be honest I feel that any amount of learning about computer architecture and vector fields is unhelpful if you are trying to understand good eval benchmarks or prompt engineering techniques.
Personally I like to learn the foundations but there's genuinely room for useful knowledge of SOTA techniques even without the foundations. To be honest I feel that any amount of learning about computer architecture and vector fields is unhelpful if you are trying to understand good eval benchmarks or prompt engineering techniques.
> unhelpful if you are trying to understand good eval benchmarks or prompt engineering techniques.
You are absolutely correct. I jumped to conclusions when I saw the list and read "AI Engineer". The reading list isn't addressing people who want to build AIs, but those who want to maximize and optimize their results with the existing ones.
My bad.
You are absolutely correct. I jumped to conclusions when I saw the list and read "AI Engineer". The reading list isn't addressing people who want to build AIs, but those who want to maximize and optimize their results with the existing ones.
My bad.
what is ELIZA? github link?
The reading list is old about one year, for instance in 2025, one may use KTO for math, RLOO for CoT, DPO for function calling and optimization.
In 2025 one should only focus should be distillation & optimization.
In 2025 CoT is not new, the corrected CoT is the key and all you need.
In 2025 one should only focus should be distillation & optimization.
In 2025 CoT is not new, the corrected CoT is the key and all you need.
Awesome list!
Am interested
[1]: https://www.bishopbook.com [2]: https://www.oreilly.com/library/view/ai-engineering/97810981... [3]: https://d2l.ai [4]: https://udlbook.github.io/udlbook/