HackerLangs
TopNewTrendsCommentsPastAskShowJobs

impossiblefork

2,125 karmajoined vor 5 Jahren

Submissions

Reversible Deep Equilibrium Models

arxiv.org
3 points·by impossiblefork·vor 10 Monaten·1 comments

comments

impossiblefork
·vor 5 Tagen·discuss
A system is responsible for anything that happens when it is permitted to operate.

You can't separate the EU from what people use it to do.
impossiblefork
·vor 5 Tagen·discuss
It's strongly connected to the structure of the EU though, and the weak control that voters have over appointments to the commission, and every level of indirection is one at which the appointer can be influenced.

If EU institutions are used to push this sort of thing, we must treat that as what they are for. Systems do not get a pass because someone external is 'using them', but must be treated holistically.
impossiblefork
·vor 7 Tagen·discuss
Left-leaning politics is not at all like early 1900eds left-leaning politics.

Left-leaning politics has moved to very mild, not-even-social-democracy policies, taxation of wage income, a decreased focus on capital owners.

Left-leaning politics has thus been transformed beyond belief and has very little to do with what it used to. Most politicians have no idea about physical reality, which is the ultimate source of technology, but live sometimes in a world of administration, sometimes in a world of laws and sometimes in a world of politics only.

Left-liberals don't exist. Liberalism is a right-wing ideology: free trade, laissez-faire.

So I don't understand at all what you mean. What are the SocDems who have gone from being SocDems to not knowing what social democracy is and who now think about things like welfare and administrative stuff and living in a world of compromises attached to?

I can't see that they're attached to anything, and I think I despise them for it. At least someone who looks back to the past can look at it and critique it and see what ideas were valuable, what the real goals were, that led to different positive achievements.
impossiblefork
·vor 9 Tagen·discuss
I wouldn't say that making the matrix diagonal in some basis is some further step.

If we have an singular value decomposition, M=USV^*, the columns of U are linearly independent they are a basis for the space M maps things into, and the columns of V are linearly independent then it's a basis for the space it maps things from, and [M]_{BB'} = S.
impossiblefork
·vor 10 Tagen·discuss
>That's you. but nobody In Sweden drives to work?

A smaller fraction than in the US. I think most people I know drive.

>I see walking to work as an relative to each individual and their job lcoatiopna dn circumstance of where they live, not a country related thing.

Well, it isn't. It's about how walkable environments are.

>GDP growth "experts" would disagree. It's the reason we don't have mandatory WFH for white collar jobs after Covid proved it's possible and salves the environment

Well, they may disagree, but the whole point is the goal of society isn't GDP, since GDP is easy to game with things like creating situation where people are effectively forced to waste energy, drive to work-- that sort of thing.
impossiblefork
·vor 10 Tagen·discuss
Yes, but German society is structured to require much less energy, just as Dutch society is structured to use much less land.

If you put Germans whose lives function in a US-style, even just getting to work will be a huge drag.

Misery depends on the structure of society. Here in Sweden I can walk to work. This means that I'm spending zero money on travel to work, and that my travel to work contributes $0 to Swedish GDP. But this is actually better than if Swedish GDP were higher and I was traveling by car.

This is one way in which GDP can be extremely misleading.
impossiblefork
·vor 12 Tagen·discuss
Ah, yes. That's good too. Especially VSORA and the interconnect, and I guess the CPU is needed too. I guess what remains is the training chip and OpenChip will be the first.
impossiblefork
·vor 12 Tagen·discuss
A lot actually. Obviously not to write these comments, but a whole lot of Claude.

With regard to the substance: sales of ASML machines are not currently connected to the EU getting chips, but to ASML getting paid for their work. For EU chips for training transformer models we'll need chip design firms, not chipmaking, and as I stated in my comment, there are some promising ones that will probably be able to design the chips we need if we order them.
impossiblefork
·vor 12 Tagen·discuss
Yes, but what does that affect?

If we end up with a world where only US firms can use the latest LLMs and the latest LLMs are needed to keep up in the software world, or in making prototypes, then that's a whole series of fields which are blocked from us.

So I think we need to make sure that we no only can, but build frontier LLMs from scratch-- not RLed on foreign data, but genuinely from scratch.

Even from an economic point of view, I don't think a continuous outflow of ~200 USD/month for every office job is sustainable, and that's what we'd get if the plausible scenario is borne out. An inter-EU cost of 200 USD/month for every office job though, that's survivable.
impossiblefork
·vor 12 Tagen·discuss
I think the way to do it is this: let EU chip design firms bid. They say what their systems can do, they give their prices, and then we choose the one that can achieve the requirements (pretrain multiple 10T+ models in a reasonable amount of time and then do RL on them) at the lowest cost.
impossiblefork
·vor 12 Tagen·discuss
Yeah, but if its final performance comes from being trained with data from a bigger model one can question whether it's a way to build genuinely new 40B models.
impossiblefork
·vor 12 Tagen·discuss
Well, some risk must be taken, even the risk of such things.

Some sort of alternative must be created, after all, since models are being restricted to the US only.
impossiblefork
·vor 13 Tagen·discuss
Yes, of course people would be working there for the paycheck.

I think there's nothing special about public funding though. The field is so competitive that people will be mad if a competitive model is not achieved, making corruption more damaging to the organizations. There would also be some internal competition. There are after all several EU LLM/AI/etc. firms that would probably try to use this infrastructure.
impossiblefork
·vor 13 Tagen·discuss
I think the chips alone are 10B minimum. It'd be way bigger than CERN.

Provided that the systems work, they can at least be repurposed to other things. If the organizations that are to train public LLMs can't do it, we can rent the system out to Mistral or something.

So I think something like 5B, starting with 10B to get started, in public money per year, the chip firms are private, some of the LLM firms will be private, but the system is available to train European LLMs-- that's I think a realistic approach.
impossiblefork
·vor 13 Tagen·discuss
I think what's actually needed is two things: an EU training infrastructure that allows training of 10T+ models, and an EU inference infrastructure that is sufficient that it's possible to do RL on them.

This effectively reduces the problem to a specialized supercomputing infrastructure problem which I think is relatively easy to solve. I think the chips are coming. I think Euclyd will be able to do the inference chip and I think the training chip won't be harder. It's just a matter of accepting the need to order a huge number of them, being willing to think a little bit like the kind of people who operate corners. So we can be there next year, I think. What we then lack is a training chip-- maybe OpenChip can do it, maybe they can't, but there are reasonable but still unfinished projects. Maybe if Euclyd finishes an inference chip in 2027 we can have the state pay them to make a training version, put in fp32, put in communication tiles. If their design is real and works (which it should, since it's basically a fancier version of Groq, as it's described, and since even Groq works) I think the advantage these chips is likely to have would be enough that a training version would be NVIDIA-beating.

We probably need some solution for the data-- i.e. to allow people to do things that are against copyright law in a limited way, but I think it's a better idea to start EU firms than to try to attract Anthropic.

Because of the need for capital the hardware-software carousel is necessary. We can't pay for NVIDIA chips and then have NVIDIA feed that money into US firms. We have to feed money into EU chips that either carousel the money into EU AI firms or who just offer cheap chips.
impossiblefork
·vor 13 Tagen·discuss
There may be an advantage to be able to use all available data.
impossiblefork
·vor 14 Tagen·discuss
Written laws, passed by congress and senate?
impossiblefork
·vor 14 Tagen·discuss
I wish the EU were legalistic and rules based, but the commission and politicians are involved in many of these things. It's like Trump's executive stuff, just with a committee instead of a single person, and I guess, with less power.
impossiblefork
·vor 16 Tagen·discuss
Robots.txt are also ToS of sorts.
impossiblefork
·vor 18 Tagen·discuss
Yes. I am Swedish too.

Notice again the lack of warning shots. The courts really want you to do one.