Ok I think I need to go into more depth on the examples.
I think HN knows that anyone can prompt LLMs. I do think its interesting though that this has allowed PMs/SMEs to direclty influence products that are deployed to millions of people. That seems genuinely novel. Maybe I over egged it
wrt did you read the article? I was quite specific about the ways I think LLMs are blurring the lines. I don't think its true for general engineering but I do think its true for applications being built with LLMs.
I agree with that. What do you think about the point thought that for LLM agents and applications, prompts and tool definitions might matter more than code?
I totally agree that we're not at a point where AI can write most code. Though, I didn't ever say that. I just think its blurring the boundary between engineers and PMs with both taking on more of the others role.
Also, it shouldn't be surprising that the product we're building is aligned with what we believe about the world :)
I think I worded this poorly. What he said was that a lot of people say they want open-source models but they underestimate how hard it is to serve them well. So he wondered how much real benefit would come from open-sourcing them.
I think this is reasonable. Giving researchers access is great but for most small companies they're likely better off having a service provider manage inference for them rather than navigate the infra challenge.
I know that HN likes to nerd out over technical details so thought I’d share a bit more on how we aggregate the noisy labels to clean them up.
At the moment we use the great Skweak [1] open source library to do this. Skweak uses an HMM to infer the most likely unobserved label given the evidence of the votes from each of the labelling functions.
This whole strategy of first training a label model and then training a neural net was pioneered by Snorkel. We’ve used this approach for now but we actually think there are big opportunities for improvement.
We’re working on an end-to-end approach that de-noises the labelling function and trains the model at the same time. So far we’ve seen improvements on the standard benchmarks [2] and are planning to submit to Neurips.
Humanloop | Infrastructure for AI | Backed by YC and Index | London + Remote
Hiring
- Software Engineers: front-end specialist
- Machine Learning Engineer
- Interaction designer
(see jobs.humanloop.com for full details)
We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there.
ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible.
No, not 100% accuracy. I've left out details for the sake of brevity but with a precision and recall high enough for the team to be able to answer the questions they cared about.
I think the mistake of this quote is in the application of the expertise. The bitter lesson is that data + compute can outperform inductive biases but that doesn't mean you don't need domain expertise to get the right data.
I think this is one of those points that is obvious in retrospect but almost universally under appreciated.
Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks.
Perhaps it should be obvious but its definitely over looked in much of academic ML and in MLops.
Hi Andy,
thanks for the feedback on the site! We're actually redesigning at the moment so it should hopefully be fresher soon :P.
Also great pointer to Rob Munroe's book. He actually used to be CTO at figure 8 before they were acquired.
You seem to be pretty clued up on the area, what do you see as the pros and cons of an end-to-end approach?