ilaksh

10,489 karmajoined قبل 17 سنة

email [email protected] or runvnc on Discord runvnc on GitHub

comments

قبل 60 دقيقةdiscuss

Dumb question, but when the Nebius capacity dashboard says they have around 3 non-preemptible B200s available, does that mean _total_, or is it just how many I myself might be able to rent on demand?

One aspect of the profitability might be the utilization and the pricing a few years down the line for slightly older hardware. Already now it seems like the increased processing you get from newer devices versus the cost difference makes something like an H100 or even A100 significantly less desirable than newer more powerful ones. As an individual, I am happy to be able to get an H200 on demand, but the B200 or B300 can do so much more work with optimized software and models for only modestly more cost that if those become available then from a business perspective you really have to prefer that if you can keep it occupied.

Then with Vera Rubin being like 3 times more effective or whatever, that adds a new layer of gradual obsolescence. So the question is can they keep the pricing up on the older ones a few years down the line enough to fill out the end of those expected payback periods.

The real boogeyman for a neocloud that has heavily invested in expensive Nvidia hardware might be a variation of that beyond Nvidia with startups that have even more dramatic efficiency increases pushing the leading edge even further. For example, if companies like Mythic AI and d-Matrix could somehow rapidly rapidly scale, that would push prices down for all of Nvidia hardware that is significantly less efficient.

I guess so far it doesn't look like any startups with really big efficiency breakthroughs are even close to being able to scale like Nvidia though, especially with the manufacturing and power crunch. But I suspect some of that is because of favoritism and strong arming protecting investments rather than a free and fair ecosystem.

قبل ساعتينdiscuss

Luckily there is nothing about the current period that reminds anyone of Hitler's time.

AI discussions these days remind me of college when I agreed to "debate" some Christians. You can assign the camps however you want to assume. But the point is that there is not going to be a lot of constructive discussion.

قبل 3 أيامdiscuss

What are the possibilities for adding more high level tasks like "pick up the [arbitrary thing]"? I assume it's 100 times harder to deal with hands and arms in a generic way. But maybe for grippers with two claws it could be more tractable to just output two force vectors per claw or something for the grasp and another two fir the drop. And maybe the SDJ could do reverse kinematics or something.

But one RGB image wouldn't work. So maybe one would need a depth camera.

قبل 3 أيامdiscuss

Zulip has been open source since 2015.

قبل 3 أيامdiscuss

From what I can see, the license for this (AGPL) is more restrictive than Zulip (Apache 2). So I would stick with Zulip.

قبل 3 أيامdiscuss

if you don't find that then you could fake it with personaplex possibly but making another ASR/STT model just listen continuously and transcribe then send to an LLM with function calling. at least that would allow one direction easily.

قبل 3 أيامdiscuss

Are there any open source full duplex models that are out besides PersonaPlex? There was a chinese open one, maybe Fun Audio chat or something, that said it was going to release a full duplex version but I am not sure if it did.

My dream would be open source full duplex with function calling or some kind of rudimentary text output. PersonaPlex is still interesting although it was looking like we would need to fine tune it to handle outgoing or avoid going off the rails easily.

قبل 5 أيامdiscuss

Well, Taalas has that kind of technology, but the chip they demoed is probably 20-100 times smaller than necessary since it's only an 8b model.

But let's say they could someday scale that up to a much larger model, 72 large chips per wafer and each chip can do 1000 LLM requests at once (Vera Rubin?). So it's roughly the equivalent of an NVL72 rack.

You might be able to serve something like 50000-60000 requests at once. So I think it's more like handling a small city's worth of customers per wafer than the world if you had that.

I believe in less than 5 years we will get to that, but the model size and/or number of agents is going to keep going up also.

قبل 5 أيامdiscuss

I think the profits depend on how well they manage their fleet purchases (or possible sub-leasing?) to get high utilization without overloading or idle racks.

Because accelerators like H200, B300 etc. are highly parallel and designed to run like 200 or maybe 300 sequences at once (depends on the model, just guessing). I assume they finance the hardware and that cost per device or rack is the same whether each unit is handling 10 requests or 150 requests (aside from electricity).

And probably international customers factor into it to get good utilization over more of the night time. And it likely is something that they look at quarterly more seriously than monthly. The biggest risk to profits might be a downturn in business that causes some portion of the financed AI accelerators to go idle or get low utilization for some weeks (that they can't sublease).

قبل 6 أيامdiscuss

Shocking that a well executed AI tutor improves outcomes.

Hasn't computer assisted interactive learning already been proven for years? Why does there seem to be so much skepticism about enhancing it with AI?

Is this just something like, astoundingly slow adoption or poor execution? Being held back by paper textbook makers? Teachers unions dragging their feet?

How can interactive AI driven individually paced learning _not_ be obviously dramatically more effective?

قبل 6 أيامdiscuss

They were using Sonnet 4.6 for some fre form responses so that could be applied to something subjective.

قبل 6 أيامdiscuss

That makes sense in a way, but remember that Meta had previously seen some brief developer glory in the initial Llama release. Going the off-the-shelf route would essentially be giving up on being on the technology frontier in this area, and not monetizing their knowledge assets.

قبل 6 أيامdiscuss

If that is backed up by benchmarks then maybe they should imitate whatever Cursor did. What did they do?

They may eventually have to do that. Or they might be starting with an existing Llama model. Maybe I should have said "huge breakthrough or additional dataset".

قبل 6 أيامdiscuss

My instinct (for better or worse) is usually contrarian. Most people seem very skeptical of what Meta is doing with AI. But, what if, in a way at least, it makes sense?

Maybe Wang has correctly identified that the programming and agentic ability that Anthropic and OpenAI models have has largely come from armies of software engineers creating massive datasets by writing out coding and agentic problems and solutions?

So he told Zuckerberg that. The reason it may be turning into so much friction is that at companies like Anthropic or OpenAI, training engineers were either hired specifically for that purpose or probably mostly handled through contracts with third parties (which again, hired them to train AI). And honestly many of them may be overseas or just happy to have a job in a difficult period. But anyway they wouldn't have very high salary expectations etc.

But Zuckerberg already had 25000 engineers. Why not take say 1/5 of them and get them working on the the dataset? The problem is that those engineers were hired for different prestigious highly paid positions at Meta/Facebook. They were not hired to do tedious grading of AI answers or quiz construction.

But Zuckerberg either has to do this, or spend additional billions on doing it all with external contractors. A third option would be to try to create a massive distillation operation. Or just hope that his engineers could invent some magical new training trick that manifested the agentic and programming skills without the large scale human input.

Or he could release a model trained largely by existing open weights models. Which without some huge breakthrough probably has no chance of surpassing them, so is pointless.

I think most of the substantive criticism of Zuckerberg has been about burning funds. If he gives up the "your job is to grade AI homework now" plan because his engineers refuse, he would need to go through third parties. The additional billions and billions this would cost would create more pressure on the bottom line and shareholder pressure.

It would also give up any potential advantage that Wang may have optimistically sold the operation as, on that using "real" engineers as opposed to lower paid data labelling engineers might result in a higher quality dataset.

At some point, model architectures that don't need such massive datasets or can be created automatically in a way that advances the frontier will probably come about. But right now it doesn't exist.

Further, the way AI works currently, business advantage from AI comes from encoding existing internal intelligence and knowledge. Meta's massive engineering corp effectively has that in their heads. Having them create these datasets is possibly the only way to leverage this knowledge asset in this paradigm.

I guess the problem is it means forcing thousands of people to do a different job from the one they were hired for.

قبل 14 يومًاdiscuss

Which implies that you cannot reasonably serve the "disfavored" cohort (self-funded startup or small business). In other words, you have admitted that I was correct in that the business model is deliberately aimed at businesses that likely can absorb surprise expenses, and it's not appropriate for others.

قبل 14 يومًاdiscuss

Any chance they will have this for Qwen 27 b also?

قبل 15 يومًاdiscuss

So write "WARNING: your service will go down if you exceed this limit. We cannot provide refunds for this. To avoid outages in the event of unanticipated traffic, do not enter a spend limit."

قبل 15 يومًاdiscuss

Why is it that you claim limits are unworkable? If you can track or enforce it (others have been for years) then couldn't you make it an optional field or checkbox?

قبل 15 يومًاdiscuss

The short version is it seems like a big "gotcha" that there is no way to limit bandwidth or spending on that or other resources ahead of time, and that might be a deliberate business model that is more aimed at well-funded startups or large companies that are monitoring costs much less closely than an individual or small business.

It's not necessarily too hard to just not dynamically spawn a bunch of machines, but the bandwidth one is going to sneak up on people.