DevRel at HuggingFace(dx.tips)
dx.tips
DevRel at HuggingFace
https://dx.tips/huggingface
104 comments
Hi all! I'm Omar from Hugging Face. Happy to answer any questions you might have about Hugging Face in general, llamas, and open ML!
Could you talk more about HuggingFace's new benchmark for LLMs? When did it become obvious that the old benchmarks were no longer sufficient:
[author here] we interviewed the maintainer of that leaderboard if you want to hear from her directly! https://www.latent.space/p/benchmarks-201
tldr: old benchmarks saturated, methodology was liable to a lot of subtle biases. as she mentions on the pod, they're already working on leaderboard v3.
tldr: old benchmarks saturated, methodology was liable to a lot of subtle biases. as she mentions on the pod, they're already working on leaderboard v3.
How large is the staff at hugging face?
We have ~220 total team members across all roles
I'd love to work at Hugging Face! Happen to be hiring any new dev relations/product/AI engineer type roles?
This is such a great piece.
This is such a great piece.
It feels hard to ask this without sounding snarky, but how do you separate growth from making good decision and having a good product from growth (in this case) because AI is hot and HF offers a lot of utility for free?
well for the last point, at least, you have the fact that HF self reports profitability
Interesting. Independent of the specific strategies, I always think of HuggingFace as squandering potential. They could/should/can be the singular place where you explore _and_ productionize these open/social models. But their offerings are nowhere near as good as Replicate and the others. At least last I checked.
[deleted]
DevRel is just a transient phase during ramp-up. Once you get market dominance, you squeeze the independent developers by controlling their platform access and pricing, and keep all the high-revenue products for yourself. See Apple, Microsoft, Google, etc.
I'm curious here how do they make money? i heard that they have ai consulting service. is that sufficient to run the company? i don't think any researchers i know pay for hugging face.
Most AI startups are following the Underpants Gnomes business plan:
Phase 1 - Launch product which loses money
Phase 2 - ???
Phase 3 - Profit!
Few have progressed past the first phase yet.
Phase 1 - Launch product which loses money
Phase 2 - ???
Phase 3 - Profit!
Few have progressed past the first phase yet.
Hugging Face is already profitable according to Clement
https://x.com/ClementDelangue/status/1811675386368966682
That is a well known lesson. In any goldrush it is always profitable to sell pickaxes and spades to the miners. The miners themselves generally do less well (except the happy few that strike gold)
[deleted]
I’m not entirely sure but I suspect through their ML hosting/inference/training products and services they have working with enterprises. But I’m still baffled how they cover their AWS storage and egress fees :)
I remember seeing a post last week-ish from Julien Chaumond that they are profitable.
I remember seeing a post last week-ish from Julien Chaumond that they are profitable.
I was curious as well what the business model is.
They are selling related services. Looks like data and inference offerings with per seat pricing. https://huggingface.co/pricing
They are selling related services. Looks like data and inference offerings with per seat pricing. https://huggingface.co/pricing
I had a similar idea for something like hugging face in 2016. (I feel so stupid I didn't execute)
My business model was that people with great models trained on proprietary data sets would buy & sell trained weights and fine tuned models for private use.
Here is an example.
No e-commerce business is going to easily publish their sales data for everyone to train so they can build a great model for optimizing recommendations of products.
Hiring an internal AI team may have 2 problems.
- It may cost too much - May not work as well - You may not have the appropriate data set or data size (DL is very data intensive).
For a small fraction of what it may cost to hire your own team. You may buy a fine tuned models or readily trained weights that you just "plug & play".
That sounds like reasonable model.
The client (e-commerce) saves money, time, is guaranteed success, and gets the bonus productivity output from AI integration.
What do you guys think? Tarpit idea?
If I owned hugging face, I would follow such a strategy.
My business model was that people with great models trained on proprietary data sets would buy & sell trained weights and fine tuned models for private use.
Here is an example.
No e-commerce business is going to easily publish their sales data for everyone to train so they can build a great model for optimizing recommendations of products.
Hiring an internal AI team may have 2 problems.
- It may cost too much - May not work as well - You may not have the appropriate data set or data size (DL is very data intensive).
For a small fraction of what it may cost to hire your own team. You may buy a fine tuned models or readily trained weights that you just "plug & play".
That sounds like reasonable model.
The client (e-commerce) saves money, time, is guaranteed success, and gets the bonus productivity output from AI integration.
What do you guys think? Tarpit idea?
If I owned hugging face, I would follow such a strategy.
[deleted]
They sell subscriptions to their products for teams and collaboration (dont ask me why its needed tho)
I mean, it's also the fact that they were the first to create a hub for the most massive secular technology trend since the advent of the internet. It's almost like anything short of utter incompetence would have worked here. So as interesting as it is to hear how they did it, it's not necessarily actionable advice or the most efficient, effective, productive approach
This seems like survivorship bias. Kaggle was pretty well-positioned to do what Hugging Face is doing (and in fact, they are trying now somewhat). There's also a lot of others that tried to build a marketplace for AI models and failed.
Kaggle is trying too many things at the same time. It's confusing. If you confuse, you lose.
Kaggle always was a place to conpete in data science contests to me. I never associated it with anything else. If they did sell ML models they did a horrible job letting people know about it
I don't disagree. My point is that Hugging Face did what others could have done. I don't think it was just luck.
I think you’re underplaying HF by a significant margin. There were many opportunities to blunder along the way (and there still are btw).
Lucky to be part of this company and seen this strategy work close up. One thing I'll add is this "decentralized" approach applies to all Hugging Face teams, not just the developer advocacy team. Just to give an example, there's no central comms team at Hugging Face, every team (usually the engineers who work on the product or features) does their own comms across the channels they think work best. That means there's lots of experimentation and most of our hires tend to be generalists who are comfortable wearing many hats.
[deleted]
I know a pretty affluent tech grifter who attended this. I'll pass on this nonsense.
In every company I've worked for, the biggest fans/lovers of meetings/let's "touch base" have always been: The most useless members of our team.
And I truly mean that, whether PM/TPM/or even SDM/SDE. The common denominator is they were always absolutely useless, adding no value or no true technical contribution, and their only value add was siloing information and claiming to be in "meetings" to lead some effort forward.
Their entire job function could've probably been replaced by a wiki or Google Doc. They intentionally made themselves the only points of contact and did not introduce people cross team because people would probably immediately realize how useless they are.
I've seen this multiple times, multiple roles, L5+, earning 100s of thousands of dollars a year at big tech companies.
Yes, I will waste 45 minutes of my time explaining word for word to you, exactly what is written in this design document, and none of it will be documented, no AIs will come from it, so you can look "busy" for today and tell your manager you did something. Woohoo collaboration!
And I truly mean that, whether PM/TPM/or even SDM/SDE. The common denominator is they were always absolutely useless, adding no value or no true technical contribution, and their only value add was siloing information and claiming to be in "meetings" to lead some effort forward.
Their entire job function could've probably been replaced by a wiki or Google Doc. They intentionally made themselves the only points of contact and did not introduce people cross team because people would probably immediately realize how useless they are.
I've seen this multiple times, multiple roles, L5+, earning 100s of thousands of dollars a year at big tech companies.
Yes, I will waste 45 minutes of my time explaining word for word to you, exactly what is written in this design document, and none of it will be documented, no AIs will come from it, so you can look "busy" for today and tell your manager you did something. Woohoo collaboration!
I, for one, represent this comment.
Hard to say what situations you have experienced because the details are scant, but in my experience usually those people have helped, in one way or another, to ensure that my pay check has the specific number that it does.
Simply put: they use the information they gain to justify the utility of individual team members to the overall effort and thus the proportionate compensation.
Simply put: they use the information they gain to justify the utility of individual team members to the overall effort and thus the proportionate compensation.
No my friend, the people who ensure the paycheck has the specific number it does are some drones in HR far removed from you. They determine reasonable bands for your role based on industry standards and how the company wants to position itself in the market.
Then your individual contribution and performance determines an additional +-10%.
Then your individual contribution and performance determines an additional +-10%.
Don't know which company you work for but this is unequivocally not true at mine. HR are subservient to Engineering (our CEO started as a senior SWE ~15 years ago). I am also one of those "useless" EMs that barter for the comp packages of my most valuable SWEs.
Your situation seems rare.
From my experience there's a lot more wiggle room in those pay bands than you'd think. Along with that, at many orgs your EM has to haggle on your behalf with their boss and every other EM that shares their budget ~yearly for your share of a limited pool of incremental dollars. I've specifically requested not to be under certain EMs in the past for that reason.
[deleted]
[deleted]
> Yes, I will waste 45 minutes of my time explaining word for word to you, exactly what is written in this design document
And they will misunderstand you, turn around and butcher the explanation to some higher up, indirectly claiming credit for your work as the messenger of its result. If something goes wrong with the project, they'll definitely make sure to emphasize your involvement in the failed initiative, strategically distancing themselves.
And they will misunderstand you, turn around and butcher the explanation to some higher up, indirectly claiming credit for your work as the messenger of its result. If something goes wrong with the project, they'll definitely make sure to emphasize your involvement in the failed initiative, strategically distancing themselves.
This reminds me of the saying https://en.wiktionary.org/wiki/success_has_many_fathers,_fai... that I’ve seen often played out in big companies, too.
[deleted]
I don't think meetings or OKRs are inherently bad, I have worked in organisations where these kinds of things are actually done well. Equally no OKRs and no meetings can be fine as well.
Where the problem exists is when the systems and processes business use start to become calcified into the organisation's way of working, and compliance is used to control the work people do.
RACI, RASCI, DACI as an example are all fine mental models for thinking about how people or teams should work on a project. But if I'm planning a project and propose a DACI, and then my project ends up getting derailed because some other manager thinks we should use a RACI instead it's probably game over.
Where the problem exists is when the systems and processes business use start to become calcified into the organisation's way of working, and compliance is used to control the work people do.
RACI, RASCI, DACI as an example are all fine mental models for thinking about how people or teams should work on a project. But if I'm planning a project and propose a DACI, and then my project ends up getting derailed because some other manager thinks we should use a RACI instead it's probably game over.
The kind of momentum maintaining alignment can provide is unreal. I hope this topic and HuggingFace and other doing it keeps getting more attention and details being shared. Nothings perfect, but it sure seems like it would work well for some people :)
Having a team of thoughtful and professional self-directed learners and creators sounds amazing if you can get a space for a team to come together around it.
Companies that have to, or choose to compromise on their labour force, either from a competency or salary perspective often have to put more supports in place to keep people longer.
Doesn't mean OKRs, etc are the correct solution.
Having a team of thoughtful and professional self-directed learners and creators sounds amazing if you can get a space for a team to come together around it.
Companies that have to, or choose to compromise on their labour force, either from a competency or salary perspective often have to put more supports in place to keep people longer.
Doesn't mean OKRs, etc are the correct solution.
Their entire job function could've probably been replaced by a wiki or Google Doc.
I'm an Engineering Manager these days and I suspect this is true to an extent for my role. All I seem to do is relay information from one group to another, translate from one jargon to a different jargon, figure out who to talk to in order to get someone to be able to push a button, and, more often than I'd like, sit in meetings and remember things when someone else has forgotten. If my teams and the teams they work with wrote things down in discoverable and organized ways I would be out of a job.
However, due to the fact my teams are made up of fallible, disorganized, and very human people, my job seems pretty safe, and I get a great deal of satisfaction from knowing I help a lot of people get a lot more done than if I wasn't around.
I'm an Engineering Manager these days and I suspect this is true to an extent for my role. All I seem to do is relay information from one group to another, translate from one jargon to a different jargon, figure out who to talk to in order to get someone to be able to push a button, and, more often than I'd like, sit in meetings and remember things when someone else has forgotten. If my teams and the teams they work with wrote things down in discoverable and organized ways I would be out of a job.
However, due to the fact my teams are made up of fallible, disorganized, and very human people, my job seems pretty safe, and I get a great deal of satisfaction from knowing I help a lot of people get a lot more done than if I wasn't around.
I'll give you a big reason why this behavior exists: people refuse to write high quality documentation regarding their systems/processes/etc. Or they are willing do it, but do an extremely poor job and are oblivious to its low quality. This will necessitate meetings to "Clear things up". And sometimes your documentation ticks all the boxes, but people get pissy about updating it because it's not in their preferred format and it rots away into irrelevance.
When I became an engineering manager I invested a ton of effort into providing solid documentation where people expect it: Swagger docs with real examples for our RESTful services for devs, confluence documentation for wider audiences, various guides and FAQs, and even in-repo ADRs. It probably saves us about 15 labor hours of meetings a week and maximizes my team's hands-on-keyboard time and minimizes interruptions.
When I became an engineering manager I invested a ton of effort into providing solid documentation where people expect it: Swagger docs with real examples for our RESTful services for devs, confluence documentation for wider audiences, various guides and FAQs, and even in-repo ADRs. It probably saves us about 15 labor hours of meetings a week and maximizes my team's hands-on-keyboard time and minimizes interruptions.
It’s kind of crazy you’re describing the person that made me leave tech to the last detail
The weird part to me is I’ve worked with PhD folk, DARPA hackers, FAANG folk and they were so kind to me even when I said dumb things the person on the other hand was just idk how to describe it made coding just painful idk why though to this day
The weird part to me is I’ve worked with PhD folk, DARPA hackers, FAANG folk and they were so kind to me even when I said dumb things the person on the other hand was just idk how to describe it made coding just painful idk why though to this day
> "growing from 780k to 2.3m repos on the HF Hub in the past year"
Is this what caused the title to be "300%"?
a. that's actually ~200%
b. isn't growth, as defined by number of repos (which might be free) the wrong metric. It's super is to scale adoption when you're giving something away for free
Is this what caused the title to be "300%"?
a. that's actually ~200%
b. isn't growth, as defined by number of repos (which might be free) the wrong metric. It's super is to scale adoption when you're giving something away for free
I was going to correct you but I was wrong
For anyone else that got tripped up... 1x to 3x is 200% growth because you don't get to count the original 1x
For anyone else that got tripped up... 1x to 3x is 200% growth because you don't get to count the original 1x
It's kind of confusing because it depends on how you phrase it with percentages.
"The number of cores in the PC went up by 50%" means "there are now 1.5x as many cores"
However, "the number of cores in the PC is 50% what is was last year" means "there are now .5x as many cores"
In this case 200% is fairly unambiguous, but imagine a phrase like "scaling community 50% per year" which could mean different things depending on the direction of the scaling... I think it's more intuitive to use a multiplier: "scaling community 3x bigger per year" makes it easy to visualize an online community tripling in size every year.
"The number of cores in the PC went up by 50%" means "there are now 1.5x as many cores"
However, "the number of cores in the PC is 50% what is was last year" means "there are now .5x as many cores"
In this case 200% is fairly unambiguous, but imagine a phrase like "scaling community 50% per year" which could mean different things depending on the direction of the scaling... I think it's more intuitive to use a multiplier: "scaling community 3x bigger per year" makes it easy to visualize an online community tripling in size every year.
[OP here] yeah youre right i've adjusted to 200%, ty
They said 300%, not growth by 300%
IMO I prefer the former ("growth factor") calculation to the "growth by" calculation as it's simpler, it's just [the_percentage]/100 * current_value
in the other case it is (the_percentage + 100)/100 * current_value which takes an extra CPU cycle of my brain
IMO I prefer the former ("growth factor") calculation to the "growth by" calculation as it's simpler, it's just [the_percentage]/100 * current_value
in the other case it is (the_percentage + 100)/100 * current_value which takes an extra CPU cycle of my brain
[op here] i've just switched it to 200% bc its the smaller number, but i agree that was what i was thinking
(b) growth is still the right metric imo. for example, lots of open-source libraries (which by definition are available to use for free) measure and optimize for growth
[deleted]
What is an OKR?
update: dead serious. I searched. It says "objectives and key results." But what does one look like? Is it a document?
update: dead serious. I searched. It says "objectives and key results." But what does one look like? Is it a document?
Sorry you received unhelpful sass from other commenters. It stands for "Objective and Key Result" - it's a common "business-speak" initialism that basically translates to "a measurable, data-driven goal".
I'd say it's more than that. OKRs are a concrete framework that came out of Intel and was popularized by Google. But yes, the focus is on setting objectives backed by measurable results.
Can be a document, an app, whatever. One flavor is objectives are set at the top, their KRs are translated into objectives for the next management layer with their own KRs, and so on. It's an iterative process with some combination of top-down and bottoms-up activities. And afterwards, the results are reviewed. There's more to it, but that's the gist.
Can be a document, an app, whatever. One flavor is objectives are set at the top, their KRs are translated into objectives for the next management layer with their own KRs, and so on. It's an iterative process with some combination of top-down and bottoms-up activities. And afterwards, the results are reviewed. There's more to it, but that's the gist.
An objective is a qualitative goal. The key results are generally quantitative numbers that can help you measure if you are achieving that goal.
An example: an objective might be "improve application performance". The key results for that objective might be "reduce average page load time from 3s to 1.5s" and "reduce API response times by 60%".
An example: an objective might be "improve application performance". The key results for that objective might be "reduce average page load time from 3s to 1.5s" and "reduce API response times by 60%".
Got it. I guess that was just illustrative, though, because why isn't "reduce average page load time from 3s to 1.5s" the objective?
Ah, I see. It bubbles. Explained by https://news.ycombinator.com/item?id=40979799
Thanks everyone.
Ah, I see. It bubbles. Explained by https://news.ycombinator.com/item?id=40979799
Thanks everyone.
If you were to tell your CEO you're reducing page load time, they will rightfully ask you why and what problem you're solving. The objective is the simple sentence that makes the "why" clear. It might itself be subordinate to a higher level objective. The idea is that all work is clearly aligned to specific goals in a very transparent manner.
It's actually a very solid framework, but as the many comments indicate, poor implementations abound (also see Agile, DevOps, etc).
It's actually a very solid framework, but as the many comments indicate, poor implementations abound (also see Agile, DevOps, etc).
Let's say your day job is providing a website on which paying customers can upload images.
Youre currently only supporting jpeg and png.
Your objective for a quarter could be to broaden support.
So you'd plan to add 10 additional formats. If you add 8, you've achieved your goal. Makes sense, right?
Honestly, the snide comments are spot on. It was even dumber then SMART SCRUM, aka development via waterfall, which is totally agile xoxo
Youre currently only supporting jpeg and png.
Your objective for a quarter could be to broaden support.
So you'd plan to add 10 additional formats. If you add 8, you've achieved your goal. Makes sense, right?
Honestly, the snide comments are spot on. It was even dumber then SMART SCRUM, aka development via waterfall, which is totally agile xoxo
I swear this is entirely earnest. That example works. Except:
>So you'd plan to add 10 additional formats. If you add 8, you've achieved your goal.
My plan was to add 10, and I added 8. I achieved my goal?
>So you'd plan to add 10 additional formats. If you add 8, you've achieved your goal.
My plan was to add 10, and I added 8. I achieved my goal?
In the original envisioning, yes; typically, 100% was set at a stretch goal, with ~70% being the more actual target. >= 70% was thus successful.
But, most of the places I've seen doing "OKRs" are just appropriating the word but not the actual system.
But, most of the places I've seen doing "OKRs" are just appropriating the word but not the actual system.
The point is that you set a goal and made progress towards it. Ideally you choose goals for yourself and it's expected that you won't complete them all.
I still don't like it.
I still don't like it.
My experience of them was that upper management set "Objectives" like "1% YoY Online Sales Growth" for your group or team or whatever and then each team in that group (again, or whatever) would come up with their own "Key Results" that they thought, if the result was achieved, would translate to that objective.
So for example, if you worked in online retail like I did, maybe you'd get an objective like that and then hypothesize a few things along the lines of "if we increase product image interactions by X% then sales should go up by Y%." as key results that you then report back out to management to show progress towards your objective.
So for example, if you worked in online retail like I did, maybe you'd get an objective like that and then hypothesize a few things along the lines of "if we increase product image interactions by X% then sales should go up by Y%." as key results that you then report back out to management to show progress towards your objective.
astrology charting for MBA’s solely because Google did it at one point
Bless your heart
https://youtu.be/XAeKtyL2m-Q [Interview with Jr. Product Manager]
A buzzword that no one actually takes seriously and means very little in practice but that managers use to justify their jobs.
The biggest impact I've seen is that you cannot adjust priorities for an entire quarter.
I've sat by and watched an entire year elapse while we were forced to dispassionately pursue an OKR we knew was leading us nowhere.
[deleted]
Why is it that whenever a company happens to be successful, and does meetings/goals differently than most (also successful) companies, the success is attributed to the lack of meetings/goals, and not, y'know, the business?
[deleted]
Being the GitHub of AI models is like the thing to be for the last two years when it comes to scaling. I'm surprised it was only 300% not like 2000%.
Not trash talking them, just agreeing that there are other factors.
Not trash talking them, just agreeing that there are other factors.
I read your comment as:
> Why is it X and not, y'know, X?
Is there something central to what a business is besides the way communication happens within it?
> Why is it X and not, y'know, X?
Is there something central to what a business is besides the way communication happens within it?
Yes. They make things that people will pay for.
I think there are a few reasons for this. The primary one is that it's a visible signal, and people make attribution errors almost constantly so it gets fixated on. I think the other reason might be that it's unpopular (and not actionable) to say "this company succeeded because the people there are better than you" or "this company had great product market fit, and the employees efforts really didn't make the difference"
Step 1: Be at exactly the optimum point during a massive opportunity.
Step 2: Do almost anything you will still grow.
Step 2: Do almost anything you will still grow.
[deleted]
this is the most accurate answer unless you are AWS, Azure, or GCP. source: ex-AWS GTM
Step 1: Buy lottery ticket with correct numbers on it.
Step 2: Cash it in and make a lot of money.
Step 2: Cash it in and make a lot of money.
This suggests the iterated optimum strategy is to just keep chasing new points trying to land that golden moment, though. If you think such points and moments are plentiful enough that you'll find them in your lifetime reliably (I do!).
> This suggests the iterated optimum strategy is to just keep chasing new points trying to land that golden moment, though.
Yup exactly. We call that venture capital
Yup exactly. We call that venture capital
Nowhere in the article it is stated what devrel stands for. What is it?
Developer relations
And what is that exactly? Is it just Jen from the IT crowd?
No, it's marketing, education, "evangelism", and other outreach focused directly towards developers.
https://en.wikipedia.org/wiki/Developer_relations https://en.wikipedia.org/wiki/Technology_evangelist
https://en.wikipedia.org/wiki/Developer_relations https://en.wikipedia.org/wiki/Technology_evangelist
> Is it just Jen from the IT crowd?
Kind of, that's not a bad analogy really. It's like if Jen knew what CI and observability were (huge over simplification but you get the idea).
Kind of, that's not a bad analogy really. It's like if Jen knew what CI and observability were (huge over simplification but you get the idea).
[deleted]
[deleted]
jtthe13(1)