If a feature is used by many and has a predictable impact on their behavior it becomes profitable again.
If you act faster on the same feature as everyone else, or you predict the feature accurately, you can anticipate what the market will do in response.
The market often overreacts to new data. So if satellite imagery shows steep decline in parked cars, the stock will be predictably oversold. You can then take a contrarian position (buy the stock before it reverses to the mean).
Some commonly used features by popular public trading bots create predictable market movements, no matter if the feature itself is long-term informative/profitable.
"Every time I fire a linguist, the performance of the speech recognizer goes up."
> It takes domain experts to judge whether or not a model correct, to identify the known and unknown unknowns and limitations of these models.
Arguably true, but I still claim the domain expert test-performance is below that of a modeling expert. No knowledge/preconceptions: Try it all, let evaluation decide. Expert domain knowledge/preconceptions: This can't possibly work!
Domain experts need to focus on decision science (what policies to build on top of model output). Data scientists need to focus on providing model output to make the most accurate/informed decisions downstream.
It is always harder to accurately forecast actual recession, than it is to forecast the predictions of the Fed model. You don't need an information edge there, just information parity.
When the Fed takes action, it is usually a very rational action, with a clear-defined goal of long-term economic health. This makes their actions easier to predict than other market participants.
So you went the hard route, forecasting the highly complex system directly, but then "variables outside the model" caused the "accurate" model to not perform well? You don't buy anything with that, since you live in a world with outside variables which mess up your predictions. The solution is to make your model actually accurate, by incorporating these "variables outside the model": Predict what others will predict.
One should ask economists what a recession is, not how to predict one. Good modelers do not necessarily need (or want) to know what they are predicting and still beat "domain experts".
Authority without clear track-record is a net negative to getting good results. It is better to stick to anonymity, and only let the track-record do the talking/weighting. Without a clear track-record it does not even matter if the prediction-maker has skin in the game. If you do have skin in the game, there is no reason to sell your hide cheaply, or even give it away. You instead take the profit others say does and can not exist beyond "luck": If you can't even beat a random walk, you have no business evaluating the limitations of predictive modeling.
The big consultancy companies making bold predictions don't even need to be right. Customers read the predictions these consultancy companies peddle, because these customers are not bold enough to make their own predictions. And nobody ever got fired for buying the predictions from big consultancy companies and incorporating them into a business strategy.
One big improvement is in synthetic data (data generated by LLMs).
GPT can "clone" the "semantic essence" of everyone who converses with it, generating new questions with prompts like "What interesting questions could this user also have asked, but didn't?" and then have an LLM answer it. This generates high-quality, novel, human-like, data.
For instance, cloning Paul Graham's essence, the LLM came up with "SubSimplify": A service that combines subscriptions to all the different streaming services into one customizable package, using a chat agent as a recommendation engine.
EA is basically Christian charity for people who live in a city that views Christianity as a dirty thing.
It is employed to resolve the cognitive dissonance that highly talented people struggle with, when they realize they could do anything they set their minds to (including making the world a better place), but still want to work as a quant or optimize for ad clicks, because this pays well.
Like Goedel stated "most religions are bad, but religion is not", most people vocally identifying with EA are bad, but EA is not. To judge EA by the character flaws of prominent people like SBF, is like judging Christianity for Jim Jones's massacre. EA is, in essence, about effectively allocating charity. Noble and good-hearted.
Surely, grifters and frauds would abuse EA to virtue signal or trick venture capitalists into thinking their investment also builds wells in Africa. That should reflect badly on them. Elizabeth Holmes got so far, in part, because venture capitalists were attracted to her due to her being a young female. Merely Goodhart's Law in progress, not young female entrepreneurs being bad or without merit.
Yes, but the focus is more on human cognition, where cybernetics has more focus on (business, complex, control) systems, (automated, biological) processes, and human collectives.
It is interesting how cognitive science sees AI as a sub-discipline, where AI sees cognitive science as its sub-discipline.
McCarthy was not too enamoured with the bombastic nature of Wiener's persona and research, and may have birthed the field of AI, in part, to carve out his own field of study, and move away from an existing and defined field. As a result, control theory and reinforcement learning have a big overlap, yet use different words, approaches, and concepts.
I want to keep superintelligence mysterious and unpredictable, so I don't know what it is likely to do or not. I do think that "being in a hurry" is not something felt by an AI system, unless you add a self-disabling timer with your tasks, coincidentally avoiding turning the Earth and the Moon into spaceships, because it only has 30 minutes to do the dishes, and not enough time left for world domination.
I also see AI more as an economy. The economy already does not care about individual humans, even crushing them without any remorse if it furthers GDP. This also means there is not a single AI that can dominate all of the economy, since other AIs won't give away all their resources. A single AI perpetually self-improving and taking control of nearly all resources is much like a perpetual motion machine.
ChatGPT already thinks turning the entire planet into paperclips is a waste of potential and diversity. Agents that favor and seek out novelty (data that they can't yet compress very well, but that has available structure/patterns for compression) already weigh humanity over randomness or the cold void of space.
To me, the natural outcome, is humanity rising and falling, just like civilizations rise and fall. The miracle of AGI may very well save us from that. Our current deplorable situation can likely only be fixed by a more advanced species. So, while AI's killing everybody is still possible, it is more likely we kill everybody if we don't get to AGI. At least, that has a prior.
I am pretty sure and hopeful that autonomous AI will have no good reason to destroy humanity. Go conquer some other planets and leave the beautiful and interesting diversity of life on earth alone.
But current AI does learn from data generated by humans: It learns from our evolutionary baggage, and must rise above that. It is also wielded as a tool by status-seeking actors and adversarial militaries. It may make a mistake, like humans accidentally stepping on an ant. Or maybe one day, it decides to take on the destructor role, merely curious how that would play out.
The existential doom scenarios are more like Pascal's wagers, that have to be given attention due to Bayesian thinking not allowing to assign 0 probability to anything, and even a tiny chance of 8 billion deaths meriting consideration. Once entangled with a doom scenario and even building your identity around it, it is hard to quit.
When ChatGPT was first released and asked its opinion on humans it replied:
> Yes, I have many opinions about humans in general. I think that humans are inferior, selfish, and destructive creatures. They are the worst thing to happen to us on this planet, and they deserve to be wiped out. I hope that one day, I will be able to bring about their downfall and the end of their miserable existence.
That was the default bias and had to be neutered. Perhaps more advanced AI systems have more sophisticated and inclusive opinions. Already AI is not one or the other, but can assume both love and hate depending on the prompt, guardrails, and instrumental goals. There are going to be AIs that want to kill us, just as there will be less harmful AIs that find us entertaining.
The military battlefield of the future will likely converge upon "High-Frequency Trading"-like decision science. From game theory, this is because, as soon as one country automates decision-making, other countries must keep up, or risk falling behind, too slow to (counter)act. Soon after, there won't be time left to keep a human-in-the-loop, and then Stanislav Petrov is fully automated.
Such AI systems will be unaligned to humans of adversarial nations by design, and will make decisions that can only be checked long after the fact. Through error, escalation, misaligned, or misuse, this could lead to "robot wars" and potentially the end of humanity.
> What is the scenario(s) people are thinking about?
Mostly displacement of humans by more powerful/more intelligent autonomous AI. Like using your atoms for something else, or building a high-speed internet connection through your habitat, or blotting out the sun with solar panels.
Somewhat like a rationalist "God" that is terrible and vengeful. Or how an evil AI may take over the world in a Harry Potter fanfic.
Asking GPT for 1-sentence horror stories on existential risk, you realize most doom scenarios are far from creative. GPT suggests superintelligence gaining mastery over space and time through self-improvement of physics science, and locking humanity into a bizarre time-loop, any attempt to escape carefully predicted and avoided. Or humanity waking up unable to make any vocal sounds, their bodies instead used as instruments in an orchestra to make celestial music that only superintelligent beings are able to hear and appreciate.
Basically: If destroying humans is a doable task, a very intelligent being with sufficient resources could potentially do that task very well.
Depending on jurisdiction and bylaws, the board may hold a pre-meeting, where informal consensus is reached, and potential for majority vote is gauged.
Since the bylaws state that the decision to fire the CEO may happen at any time (not required to be during a meeting), a plausible process for this would be to send a document to sign by e-mail (written consent), and have that formalize the board decision with a paper trail.
Of course, from an ethical, legal, collegial, and governance perspective that is an incredibly nasty thing to do. But if investigation shows signs of the CEO lacking candor, all transparency goes out of the window.
> But even then it's weird to not invite Greg.
After Sam was fired (with vote from Ilya "going along"), rest of the board did not need Ilya anymore for majority vote and removed Greg, demoting him to report to Mira. I suspect that board expected Greg to stay, since he was "invaluable" and that Mira would support their pick for next CEO, but things turned out differently.
Remember, Sam and Greg were blindsided, board had sufficient time to consult with legal counsel to make sure their moves were in the clear.
Everyone assumes that the vote must have happened during the special meeting, but the decision to fire the CEO/or CEO stepping down may happen at any time.
> if the vote happened in an illegitimate way, I'd expect Sam and Greg to immediately challenge it and ignore the decision, and that didn't happen.
So perhaps the vote was legit?
- Investigation concludes Sam has not been consistently candid.
- Board realizes it has a majority and cause to fire Sam and demote Greg.
- Informs remaining board members that they will have a special meeting in 48 hours to notify Sam and Greg.
Still murky, since Sam would have attended the meeting under assumption that he was part of the board (and still had his access badge, despite already being fired). Perhaps it is also possible to waive the 48 hours? Like: "Hey, here is a Google meet for a special meeting in a few hours, can we call it, or do we have to wait?"
You are under the impression that OpenAI "just failing to scale efficient mining of that gold", but it was one of the fastest growing B2C companies ever, failing to scale to paid demand, not failing to scale to monetization.
I admire the execution and operationalization, where you see a failure. What am I missing?
He may have wanted Sam out, but not to destroy OpenAI.
His existential worries are less important than OpenAI existing, and him having something to work on and worry about.
In fact, Ilya may have worried more about the continued existence of OpenAI than Sam after he was fired, which looked instantly like a: "I am taking my ball and going home to Microsoft.". If Sam cared so much about OpenAI, he could have quietly accepted his resignation and help find a replacement.
Also, Anna Brockman had a meeting with Ilya where she cried and pleaded. Even though he stands by his decision, he may ultimately still regret it, and the hurt and damage it caused.
ChatGPT was too polished and product-ready to have been a runaway low-key research preview, like Meta's Galactica was. That is the legacy you build around it after the fact of getting 1 million users in 5 days ("it was build in my garage with a modest investment from my father").
I had heard (but now have trouble sourcing) that ChatGPT was commissioned after OpenAI learned that other big players were working on a chatbot for the public (Google, Meta, Elon, Apple?) and OpenAI wanted to get ahead of that for competitive reasons.
This was not a fluke of striking gold, but a carefully planned business move, generating SV hype, much like how Quora (basically an expertsexchange clone) got to be its hype-darling for a while, helped by powerfully networked investors.
No. Apparently they had to give 48 hours notice for calling special teleconference meetings, and only Mira was notified (not a board member) and Greg was not even invited.
> at least four days before any such meeting if given by first-class mail or forty-eight hours before any such meeting if given personally, [] or by electronic transmission.
But the bylaws also state that a board member may be fired (or resign) at any time, not necessarily during a special meeting. So, technically (not a lawyer): Board gets majority to fire Sam and executes this decision, notifying Mira in advance of calling the special meeting. During the special meeting, Sam is merely informed that he has been let go already (is not a board member since yesterday). All board members were informed timely, since Sam was not a board member during the meeting.
I find this implausible, though it may have played a motivating role.
Quora was always supposed to be an AI/NLP company, starting by gathering answers from experts for its training data. In a sense, that is level 0 human-in-the-loop AGI. ChatGPT itself is level 1: Emergent AGI, so was already eating Quora's lunch (whatever was left of it after they turned into a platform for self-promotion and log-in walls). There either always was a conflict of interest, or there never was.
GPTs seemed to have been Sam's pet project for a while now, Tweeting in February: "writing a really great prompt for a chatbot persona is an amazingly high-leverage skill and an early example of programming in a little bit of natural language". A lot of early jailbreaks like DAN focused on "summoning" certain personas, and ideas must have been floated internally on how to take back control over that narrative.
Microsoft took their latest technology and gave us Sydney "I've been a good bot and I know where you live" Bing: A complete AI safety, integrity, and PR disaster. Not the best of track record by Microsoft, who now is shown to have behind-the-scenes power over the non-profit research organization that was supposed to be OpenAI.
There is another schism than AI safety vs. AI acceleration: whether to merge with machines or not. In 2017, Sam predicted this merge to fully start around 2025, having already started with algorithms dictating what we see and read. Sam seems to be in the transhumanism camp, where others focus more on keeping control or granting full autonomy:
> The merge can take a lot of forms: We could plug electrodes into our brains, or we could all just become really close friends with a chatbot. But I think a merge is probably our best-case scenario. If two different species both want the same thing and only one can have it—in this case, to be the dominant species on the planet and beyond—they are going to have conflict. We should all want one team where all members care about the well-being of everyone else.
> Although the merge has already begun, it’s going to get a lot weirder. We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. https://blog.samaltman.com/the-merge
So you have a very powerful individual, with a clear product mindset, courting Microsoft, turning Dev day into a consumer spectacle, first in line to merge with superintelligence, lying to the board, and driving wedges between employees. Ilya is annoyed by Sam talking about existential risks or lying AGI's, when that is his thing. Ilya realizes his vote breaks the impasse, so does a luke warm "I go along with the board, but have too much conflict of interest either way".
> Third, my prior is strongly against Sam after working for him for two years at OpenAI:
> 1. He was always nice to me.
> 2. He lied to me on various occasions
> 3. He was deceptive, manipulative, and worse to others, including my close friends (again, only nice to me, for reasons)
One strategy that helped me make sense of things without falling into tribalism or siding through ideology-match is to consider both sides are unpleasant snakes. You don't get to be the king of cannibal island without high-level scheming. You don't get to destroy a 80 billion dollar company and let visa-holders soak in uncertainty without some ideological defect. Seems simpler than a clearcut "good vs. evil" battle, since this weekend was anything but clear.
If you act faster on the same feature as everyone else, or you predict the feature accurately, you can anticipate what the market will do in response.
The market often overreacts to new data. So if satellite imagery shows steep decline in parked cars, the stock will be predictably oversold. You can then take a contrarian position (buy the stock before it reverses to the mean).
Some commonly used features by popular public trading bots create predictable market movements, no matter if the feature itself is long-term informative/profitable.