California Senate Passes SB 1047(hyperdimensional.co)
hyperdimensional.co
California Senate Passes SB 1047
https://www.hyperdimensional.co/p/california-senate-passes-sb-1047
46 comments
> What's the obvious loophole for the companies here.
As with all things under California’s broken political system: the ballot initiative process. Get people to vote for a law that in effect repeals this one, and put repealing the repeal out of reach for the legislature.
As with all things under California’s broken political system: the ballot initiative process. Get people to vote for a law that in effect repeals this one, and put repealing the repeal out of reach for the legislature.
For those that aren't familiar, this is exactly what happened with rideshare and delivery companies in California. AB5 was passed in 2019 that reclassified those workers as employees and so Uber, Lyft, Instacart, and others spent over $200m on getting Prop 22 passed in 2020 to give them an exception to AB5. Part of Prob 22 was a requirement that a seven-eighths majority in the legislature would be needed to amend it.
7/8's majority requirement??
These kind of supermajority threshold requirements should not be allowed, they're simply outrageous and blatantly obstructionist. Might was well write "this is immutable ha ha suck it"
These kind of supermajority threshold requirements should not be allowed, they're simply outrageous and blatantly obstructionist. Might was well write "this is immutable ha ha suck it"
If you can modify the California constitution by a simple ballot initiative, anything is possible within the bounds of Federal law and the US Constitution. AB5 was a bad law, but the solution was only a half-step to reversing it and yeah, the super majority requirement is essentially a bad solution to a legislature that would have had no qualms overturning a majority vote and talking a good game about how they’re right and voters are wrong.
If our political environment didn’t feature short legislative term limits and ballot initiatives to begin with and was properly representative, it’s unlikely someone like the bill’s sponsor would have either been voted in in the first place or received enough support within the legislature to get AB5 passed. She also left the legislature and found herself a nice career working for the very interest groups that wanted to see AB5 passed in the first place.
If our political environment didn’t feature short legislative term limits and ballot initiatives to begin with and was properly representative, it’s unlikely someone like the bill’s sponsor would have either been voted in in the first place or received enough support within the legislature to get AB5 passed. She also left the legislature and found herself a nice career working for the very interest groups that wanted to see AB5 passed in the first place.
>> The California government is gearing up for the “increased incarceration costs” and other costs associated with criminalizing AI development if this bill passes
>That is just so odd. Weren't they the state focusing intently on quicker rehabilitation and alternatives to incarceration but now they are preparing to incarcerate more people over AI models? What is even going on that state.
Why is this a point of contention? Isn't it obvious that any new law that defines a new type of illegal behavior is going to result in increased incarceration costs? It doesn't matter if 99.99% of people convicted of violating a new law never face incarceration. There still will be some people who will be convicted of perjury or other crimes because of this new law and dealing with those people won't be free.
This isn't an endorsement of this regulation, I'm just asking for clarification on why this specific aspect is worth singling out as a problem.
>That is just so odd. Weren't they the state focusing intently on quicker rehabilitation and alternatives to incarceration but now they are preparing to incarcerate more people over AI models? What is even going on that state.
Why is this a point of contention? Isn't it obvious that any new law that defines a new type of illegal behavior is going to result in increased incarceration costs? It doesn't matter if 99.99% of people convicted of violating a new law never face incarceration. There still will be some people who will be convicted of perjury or other crimes because of this new law and dealing with those people won't be free.
This isn't an endorsement of this regulation, I'm just asking for clarification on why this specific aspect is worth singling out as a problem.
> Why is this a point of contention?
It's ok, I guess? It just seems a little odd to go from we'll make AI model creators liable by law directly to effectively "let's expand our prison exercise yards and visitation facilities". Like they're planning on fill up the prisons right away.
On second thought, I guess, exercise yard expansion might not be quite right, I am think more like "this incarcerated population will require espresso machines and yerba mate gourds. We'll have start a commission to study how to source those...".
(One has to laugh when things get ridiculous enough...)
It's ok, I guess? It just seems a little odd to go from we'll make AI model creators liable by law directly to effectively "let's expand our prison exercise yards and visitation facilities". Like they're planning on fill up the prisons right away.
On second thought, I guess, exercise yard expansion might not be quite right, I am think more like "this incarcerated population will require espresso machines and yerba mate gourds. We'll have start a commission to study how to source those...".
(One has to laugh when things get ridiculous enough...)
I clicked through to the original source and here is the text of the note about increased costs.
>Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds of thousands of dollars to counties for increased incarceration costs relating to the expansion of felony perjury in this bill.
And the first Google result[1] says it costs $132,860 to house a prisoner in California.
So the analysis is effectively saying that California might incarcerate 0-7 people annually based off this new law. Is that supposed to be ridiculous?
This just seems like a weird aspect of this issue to highlight and it ends up making the overall argument against this bill look weaker because of course incarceration costs will go up if you incarcerate more people.
[1] - https://www.abc10.com/article/news/local/california/calmatte...
>Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds of thousands of dollars to counties for increased incarceration costs relating to the expansion of felony perjury in this bill.
And the first Google result[1] says it costs $132,860 to house a prisoner in California.
So the analysis is effectively saying that California might incarcerate 0-7 people annually based off this new law. Is that supposed to be ridiculous?
This just seems like a weird aspect of this issue to highlight and it ends up making the overall argument against this bill look weaker because of course incarceration costs will go up if you incarcerate more people.
[1] - https://www.abc10.com/article/news/local/california/calmatte...
> This just seems like a weird aspect of this issue to highlight
It's FUD from a thinktank that specializes in freaking out over corporate legislation under the guise of "personal liberty". Still seem weird?
Seems right on point to me.
It's FUD from a thinktank that specializes in freaking out over corporate legislation under the guise of "personal liberty". Still seem weird?
Seems right on point to me.
Wild California actually passed this, people joke about that state but I thought this was crazy even for them.
The biggest points imo being
>The Frontier Model Division has a wide scope to define the “perjury” charges the bill attaches to developers for end-user misconduct, including “the rigor and detail” of the developer’s safety protocols
>So is any model trained with less compute but matching a 10^26 flops-model in “relevant benchmarks” (the bill doesn’t say which)
>So is any model trained with less compute AND lower benchmarks, so long as it is of “similar general capability” (we do not know what this means)
Actually legislating that the developer of the AI is liable for what end users do is absolutely wild. And the other clauses they added make this essentially apply to any developer AI that can form a sentence. They made the categories so broad that an argument could be made that the law applies to any AI at all.
The biggest points imo being
>The Frontier Model Division has a wide scope to define the “perjury” charges the bill attaches to developers for end-user misconduct, including “the rigor and detail” of the developer’s safety protocols
>So is any model trained with less compute but matching a 10^26 flops-model in “relevant benchmarks” (the bill doesn’t say which)
>So is any model trained with less compute AND lower benchmarks, so long as it is of “similar general capability” (we do not know what this means)
Actually legislating that the developer of the AI is liable for what end users do is absolutely wild. And the other clauses they added make this essentially apply to any developer AI that can form a sentence. They made the categories so broad that an argument could be made that the law applies to any AI at all.
Just the senate. Still needs to pass the house and get signed by the governor. Not clear it will be.
the laws will more and more prohibit LLMs to produce "dangerous" information, "biological weapons" and the likes (down to the maps of public access to beaches). The information is coming from training data. And naturally that way the laws press for removing "dangerous" information from the training data. Once established as a matter of practice that you can't legally access "dangerous" info through the LLM, the natural question arises why you still can access it through the browser, and society will be mentally ready to prohibit the access through the browser too. Voila. Wet dream of the government. If you can't successfully mount frontal attack, you do flank maneuver.
Voting for new legislators, personally. I wish they'd do something about PG&E or housing instead of criminalizing software development of chatbots. Truly useless, and I wish we had more choice of non-insane candidates.
CA is essentially a one-party state with a sprinkling of GOP votes thrown in (not unlike Red states where it's flipped).
They do everything they can to reduce the pool of alternatives so they can keep the power and grift going.
They do everything they can to reduce the pool of alternatives so they can keep the power and grift going.
So, just the AI developed, used and hosted only in California, US are covered, or they are trying to legislate over all over the world? They will ban the use of non-compliant models inside the California state borders, while they will could be used freely elsewhere?
Using geography-limited legislation over things that are used countrywide or worldwide. for this or other laws from there or elsewhere is a bit overreaching. In the end, if the senate elected democratically for a territory, in representation of their citizens, decide something, is ok, as long as it only affects the people of that territory. Things gets very wrong when your elected governments impose their opinion over other territories/countries, there should a name for that.
Using geography-limited legislation over things that are used countrywide or worldwide. for this or other laws from there or elsewhere is a bit overreaching. In the end, if the senate elected democratically for a territory, in representation of their citizens, decide something, is ok, as long as it only affects the people of that territory. Things gets very wrong when your elected governments impose their opinion over other territories/countries, there should a name for that.
Cheers to OpenAI who I'm sure is popping champagne that they actually got this garbage passed. I'm genuinely shocked that legislators don't see these models as sources of information -- there's no way they would pass a law requiring libraries certify that readers can't use any of the books for evil, but here we are. And weirder still criminally liable if someone checks out a chemistry textbook and cooks meth.
These models are really cool and novel don't get me wrong but they're self-reading books on their training data and ought to be regulated as such. But hey, if it means that some other state is gonna be the hub for AI I'm all for it.
These models are really cool and novel don't get me wrong but they're self-reading books on their training data and ought to be regulated as such. But hey, if it means that some other state is gonna be the hub for AI I'm all for it.
I would nit be surprised if this is lobbied from openai, msft or google
How/why is OpenAI immune to this law?
They're not immune to the law, the law is just written so that all AI companies have to do all the bullshit safety stuff that OpenAI was already doing.
Now to be fair OpenAI's safety stuff is actually useful but it's useful for "programmable customer service agent" and companies that want to have a chatbot and not have it say fuck. But those tweaks aren't free, and come at the expense of model performance. So OpenAI pushed regulators hard on this and now everyone has to has to pay the same cost wooo!
It means you can't chip away at OpenAI's market by being better than them at specialized tasks because you have a raw completely unaligned non-instruction tuned model you use for say generating code. Because that model could be used to write viruses oohhh spooky.
Now to be fair OpenAI's safety stuff is actually useful but it's useful for "programmable customer service agent" and companies that want to have a chatbot and not have it say fuck. But those tweaks aren't free, and come at the expense of model performance. So OpenAI pushed regulators hard on this and now everyone has to has to pay the same cost wooo!
It means you can't chip away at OpenAI's market by being better than them at specialized tasks because you have a raw completely unaligned non-instruction tuned model you use for say generating code. Because that model could be used to write viruses oohhh spooky.
Okay, so they're regulating AI. What's most noteworthy to me is how applicability appears completely arbitrary and sounds like it could even be set after the fact by virtue of the jury instructions allowed.
Some domains do require regulations updates do to changes in the field, in which case it may make sense to have a commission (sorta like they're doing here), but based on this post, they basically just gave a carte blanche to disallow everything (while not directly stating that nothing is allowed).
Some domains do require regulations updates do to changes in the field, in which case it may make sense to have a commission (sorta like they're doing here), but based on this post, they basically just gave a carte blanche to disallow everything (while not directly stating that nothing is allowed).
shame on CA senate for being either too corrupt, too stupid or both for passing this. Congratulations to MSFT/OpenAI and Google for successful regulatory capture.
Sorry future startups in this field - I guess when there are enough compute in few years to start hitting this bill definitions, you will need to move elsewhere.
Sorry future startups in this field - I guess when there are enough compute in few years to start hitting this bill definitions, you will need to move elsewhere.
Wow : you can tell legislators add ad hoc laws galore, which seems so unaligned with the wisdom below.
"A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away."
attributed (perhaps not perfect translation) to Antoine de Saint-Exupery
"A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away."
attributed (perhaps not perfect translation) to Antoine de Saint-Exupery
Laws like this that can be selectively enforced are loved by power hungry people.
Any law can be selectively enforced. Laws are not programs written in code and enforced by machines.
It's much easier to selectively enforce laws that are purposefully vague.
Wait. The most advanced models have to be safe against mass monetary losses and mass casualties, but not the less advanced ones?
How exactly does that make sense?
That's like only regulating fighter jet safety while letting passenger jets do whatever they want.
How exactly does that make sense?
That's like only regulating fighter jet safety while letting passenger jets do whatever they want.
[deleted]
Oh well, gotta start somewhere I guess
[deleted]
ChatGPT and Claude can write better legislation than this.
this may make me move out of california
Where would you go?
Washington, Alaska, Hawaii, Florida, Texas …
US is a biiiiiiig country.
US is a biiiiiiig country.
This is awful.
[deleted]
As a thought experiment, I think it’s useful to try steel manning this bill, and assume for a minute that the potential risk of a sufficiently advanced and “unsafe” model is real and severe.
It seems reasonable to regulate specific aspects of safety, e.g. it shouldn’t be possible to derive the formulas for chemical or nuclear weapons. If an AI company cannot be certain of this, it seems reasonable for the government to impose restrictions on the handling of such models. Where things get murky is how to define risk for things that are less obvious.
There is deeply embedded in the tech community an ethos and belief that core tech is neutral, the potential for harmful use is an acceptable reality. I have lived and breathed this for 20 years.
But information is not neutral, and the output of these models is derived from a corpus of information that is anything but neutral. A subset of this information is so incredibly harmful that the risk of its inclusion is high enough to warrant significant effort in ensuring safety.
Again, assuming a sufficiently advanced model, might it be reasonable that the onus is on the creator of the model to understand what it can do and certify what it cannot do?
Many have commented that the non-regulation of the early Internet was necessary and good, and that the same should be true for AI. But the eras are very unlike each other. The entire technology industry was still forming, and no one knew what would come of it. What grew was incredible. But with it came the data brokers, the worst aspects of social media, etc.
It’s against this backdrop of an increasingly enshittefied landscape that regulators are looking at this potentially seismic shift in computing and not feeling willing to let the companies that cooked up the current mess go headlong into something potentially far more fantastical (for good or bad) without checks and balances.
I still think the bill is far too vague and just drives AI development out of CA. But I can understand where it’s coming from, and I wouldn’t be surprised to see a push for this nationally, with similarly vague specifications because no one knows how to quantify AI risk.
Edit: To be clear, I'm asking questions and trying to explore the legislator's angle here. I'm not saying I think this kind of legislation will be effective or that this is a good bill.
It seems reasonable to regulate specific aspects of safety, e.g. it shouldn’t be possible to derive the formulas for chemical or nuclear weapons. If an AI company cannot be certain of this, it seems reasonable for the government to impose restrictions on the handling of such models. Where things get murky is how to define risk for things that are less obvious.
There is deeply embedded in the tech community an ethos and belief that core tech is neutral, the potential for harmful use is an acceptable reality. I have lived and breathed this for 20 years.
But information is not neutral, and the output of these models is derived from a corpus of information that is anything but neutral. A subset of this information is so incredibly harmful that the risk of its inclusion is high enough to warrant significant effort in ensuring safety.
Again, assuming a sufficiently advanced model, might it be reasonable that the onus is on the creator of the model to understand what it can do and certify what it cannot do?
Many have commented that the non-regulation of the early Internet was necessary and good, and that the same should be true for AI. But the eras are very unlike each other. The entire technology industry was still forming, and no one knew what would come of it. What grew was incredible. But with it came the data brokers, the worst aspects of social media, etc.
It’s against this backdrop of an increasingly enshittefied landscape that regulators are looking at this potentially seismic shift in computing and not feeling willing to let the companies that cooked up the current mess go headlong into something potentially far more fantastical (for good or bad) without checks and balances.
I still think the bill is far too vague and just drives AI development out of CA. But I can understand where it’s coming from, and I wouldn’t be surprised to see a push for this nationally, with similarly vague specifications because no one knows how to quantify AI risk.
Edit: To be clear, I'm asking questions and trying to explore the legislator's angle here. I'm not saying I think this kind of legislation will be effective or that this is a good bill.
> it shouldn’t be possible to derive the formulas for chemical or nuclear weapons.
This information is widely available in textbooks, so...what exactly is the problem? If it's just a lossy-compression parrot "AI" like ChatGPT it won't be able to spit out anything that wasn't somewhere (implicitly or explicitly) in the training data anyway.
This information is widely available in textbooks, so...what exactly is the problem? If it's just a lossy-compression parrot "AI" like ChatGPT it won't be able to spit out anything that wasn't somewhere (implicitly or explicitly) in the training data anyway.
That is current generation (very early) tech. The hypothetical model the bill is worried about is significantly more advanced.
The key word here being hypothetical.
Getting to a significantly more advanced model will require significantly more work, and yet again we don't really know which direction to go in. Every time we crest a hill and see the ocean in the distance, we all start screaming "I see the beach! We're almost there!" but without a map we have no idea how much further we have left to travel.
Getting to a significantly more advanced model will require significantly more work, and yet again we don't really know which direction to go in. Every time we crest a hill and see the ocean in the distance, we all start screaming "I see the beach! We're almost there!" but without a map we have no idea how much further we have left to travel.
> it shouldn’t be possible to derive the formulas for chemical or nuclear weapons
Nonsense, this information is widely available and always has been. Modern chemical weapons like VX nerve agent have detailed information about synthesis on Wikipedia[0]. Any trained chemist worth a damn can replicate this. Despite this, chemical weapon attacks by non-government entities are extremely rare with only a few examples, the most notable probably being the Tokyo nerve agent attack[1].
As with nuclear weapons, the difficulty has always been in the technical execution, not the theory and know-how. And few people with the inclination to make the effort.
[0] https://en.wikipedia.org/wiki/VX_(nerve_agent)
[1] https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
Nonsense, this information is widely available and always has been. Modern chemical weapons like VX nerve agent have detailed information about synthesis on Wikipedia[0]. Any trained chemist worth a damn can replicate this. Despite this, chemical weapon attacks by non-government entities are extremely rare with only a few examples, the most notable probably being the Tokyo nerve agent attack[1].
As with nuclear weapons, the difficulty has always been in the technical execution, not the theory and know-how. And few people with the inclination to make the effort.
[0] https://en.wikipedia.org/wiki/VX_(nerve_agent)
[1] https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
The presumption is that these sufficiently advanced models are able to help close the gaps in execution. Think 20 years of advancement in both model complexity and manufacturing tech.
This is a really biased and misleading summary. In particular, it's vanishingly unlikely that anyone will have criminal penalties applied. More analysis here, going over the entire text of the bill in detail:
https://thezvi.substack.com/p/on-the-proposed-california-sb-...
https://thezvi.substack.com/p/q-and-a-on-proposed-sb-1047
https://thezvi.substack.com/p/on-the-proposed-california-sb-...
https://thezvi.substack.com/p/q-and-a-on-proposed-sb-1047
I think this law is terrible but I'm happy about it from the perspective of being able to have a state self-destruct so gloriously that it can serve as an example for everyone else.
Are Facebook AI researchers going to continue working in California when they can be put in jail? What about all the researchers in the UC system and Stanford?
Are Facebook AI researchers going to continue working in California when they can be put in jail? What about all the researchers in the UC system and Stanford?
That just sounds so strange. People will have to sign promises that their model is safe from possible end-user harm. It seems like they are setting up something for selective enforcement.
> The California government is gearing up for the “increased incarceration costs” and other costs associated with criminalizing AI development if this bill passes
That is just so odd. Weren't they the state focusing intently on quicker rehabilitation and alternatives to incarceration but now they are preparing to incarcerate more people over AI models? What is even going on that state.
What's the obvious loophole for the companies here. Leave the state? Or can they just say "This model was designed and built in Nevada/Kansas/Kentucky/etc. Us (OpenAI) are just licensing this as an end-user from our partners in Kentucky, sorry, CA attorney general, /shrug/?".