The specific mechanism of action here needs a drill-down. Did "The AI Scientist" (ugh) generate a patch to its code and prompt a user to apply it, as the screenshots would seem to indicate? If so I don't find this worrying at all: people write all kinds of stupid code all the time--often, impressively, without any help from "AI"! ;)
Did it apply the patch itself, then reset the session to whatever extent necessary to have the new code take effect? TBH I'm not really worried about this either, as long as the execution environment doesn't grant it the ability to bring unlimited additional hardware to bear. Even in that case, presumably some human would be paying attention to the AWS bills, or the moral equivalent.
Long, long ago, I used a library called Augeas (https://augeas.net/) in a drift-detection product*, so that if we detected a difference in config files, either on the same server through time, or on different servers that were supposed to be similar, we could de-noise the diffs, and more importantly, let users write fine-grained, but syntax-tolerant, allow-lists like "this particular setting is allowed to differ"... or even "this particular setting can have one of the following list of values". :)
* the company was acquired by Splunk years after we shelved that product
Building a trust relationship between commercial entities isn't automatable; it nearly always requires a contract to be carefully hand-written and argued over by high-priced lawyers before any meaningful exchange of value can take place.
Sure, this is an unfortunate level of friction, and overkill in many cases, but think about it from a cost/benefit perspective: I can spend $10k on legal fees and successfully avoid not just a lot of uncertainty, but very infrequently, the contract also protects me from losses that can be orders of magnitude larger than it cost me to negotiate the contract.