Acquisition of chess knowledge in AlphaZero(en.chessbase.com)
en.chessbase.com
Acquisition of chess knowledge in AlphaZero
https://en.chessbase.com/post/acquisition-of-chess-knowledge-in-alphazero
99 comments
> When a superhuman AI plays chess, the moves usually make sense to an expert. And even if they don't immediately make sense, they usually make sense once the expert plays out some lines to see what happens.
Do you actually play chess? Half the moves made by an AI at 3500+ ELO (e.g. Stockfish 13+) is completely alien to even a human super GM like Nakamura or Carlsen. It's actually the most reliable and used way to know whether someone is cheating or not, if the moves don't make sense to high level players, then it's very probably someone using an engine.
> And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding is still leverages the same concepts.
That is very far from being a truth that lots of chess experts would agree on. Most chess players at high level reason with similar, well established concepts, such as tempo, positions, baits, color dominance, etc.
Super AIs don't use these concepts, and it's obvious when you see their moves. They often break tempo abruptly, disregard color dominance, enter what looks like dangerous positions,etc.
Do you actually play chess? Half the moves made by an AI at 3500+ ELO (e.g. Stockfish 13+) is completely alien to even a human super GM like Nakamura or Carlsen. It's actually the most reliable and used way to know whether someone is cheating or not, if the moves don't make sense to high level players, then it's very probably someone using an engine.
> And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding is still leverages the same concepts.
That is very far from being a truth that lots of chess experts would agree on. Most chess players at high level reason with similar, well established concepts, such as tempo, positions, baits, color dominance, etc.
Super AIs don't use these concepts, and it's obvious when you see their moves. They often break tempo abruptly, disregard color dominance, enter what looks like dangerous positions,etc.
>Do you actually play chess?
Yes. I have a chess set on my kitchen counter from playing blindfold last night. I led my high school to the state championship. I am rated ~2000 in blitz and crazyhouse on Lichess. I watched 10+ analysis videos of AlphaZero when it came out.
If you have a criticism of my claim, just state the criticism. Even if I was a grandmaster, I would not be immune from being wrong.
And to be clear, my claim is that human experts usually (not always) understand the pros and cons of a move made by AlphaZero. My claim is not that human experts usually know why one move, with its set of pros and cons, was selected over another move, with a different set of pros and cons.
And of course, there are exceptions. Super long endgame sequences, for example. Even with a lifetime I doubt any human can understand these sequences beyond the tautological "it's winning because it wins": https://en.chessbase.com/post/just-one-of-17-823-400-766-pos...
Yes. I have a chess set on my kitchen counter from playing blindfold last night. I led my high school to the state championship. I am rated ~2000 in blitz and crazyhouse on Lichess. I watched 10+ analysis videos of AlphaZero when it came out.
If you have a criticism of my claim, just state the criticism. Even if I was a grandmaster, I would not be immune from being wrong.
And to be clear, my claim is that human experts usually (not always) understand the pros and cons of a move made by AlphaZero. My claim is not that human experts usually know why one move, with its set of pros and cons, was selected over another move, with a different set of pros and cons.
And of course, there are exceptions. Super long endgame sequences, for example. Even with a lifetime I doubt any human can understand these sequences beyond the tautological "it's winning because it wins": https://en.chessbase.com/post/just-one-of-17-823-400-766-pos...
Today the world's best players (not just chess) play against bots like AlphaZero. They can learn from their opponent. Today that is what it takes to be the best in the world.
Even if most moves are normal, clearly AlphaZero and other bots are doing something different or they wouldn't be world's best.
Even if most moves are normal, clearly AlphaZero and other bots are doing something different or they wouldn't be world's best.
Being friends with a bunch of grandmasters....3500 is a long ways you there. You might both be right.
> And to be clear, my claim is that human experts usually (not always) understand the pros and cons of a move made by AlphaZero.
It’s more than human experts apply a posteriori the tools they typically use to analyse chess games to games played by AlphaZero. These tools are specifically crafted to analyse chess so they obviously shed some lights on what’s happening but even then some plays were seen as counterintuitive.
An easy example is how AlphaZero can sacrifice pieces or tempos to cripple the mobility of the opposition pieces. If you carefully analyse the full game, you can see that it’s probably why the move was made but it’s completely counterintuitive when it’s played as the relative value of some winning tradeoffs made by AlphaZero are impossible to evaluate for a human player.
It’s more than human experts apply a posteriori the tools they typically use to analyse chess games to games played by AlphaZero. These tools are specifically crafted to analyse chess so they obviously shed some lights on what’s happening but even then some plays were seen as counterintuitive.
An easy example is how AlphaZero can sacrifice pieces or tempos to cripple the mobility of the opposition pieces. If you carefully analyse the full game, you can see that it’s probably why the move was made but it’s completely counterintuitive when it’s played as the relative value of some winning tradeoffs made by AlphaZero are impossible to evaluate for a human player.
> It’s more than human experts apply a posteriori the tools they typically use to analyse chess games to games play by AlphaZero.
I largely agree with this and I was about to make the same point before reading your comment. However, as GP said upthread:
> This paper builds on that concept and finds that not only are a superhuman AI's moves usually understandable by experts, but even some of the superhuman AI's internal representations are understandable by experts (!).
Internal structures mapping to concepts high level players already use is a good indicator that the AI is "thinking" about chess in the same way as humans do.
I largely agree with this and I was about to make the same point before reading your comment. However, as GP said upthread:
> This paper builds on that concept and finds that not only are a superhuman AI's moves usually understandable by experts, but even some of the superhuman AI's internal representations are understandable by experts (!).
Internal structures mapping to concepts high level players already use is a good indicator that the AI is "thinking" about chess in the same way as humans do.
> Half the moves made by an AI at 3500+ ELO (e.g. Stockfish 13+) is completely alien to even a human super GM
The analysis in the paper draws the exact opposite conclusion. Alpha zero’s play agrees remarkably with humans despite not seeing a human game (e.g. it found the Berlin defense to the Ruy Lopez).
Also your point is trivially not true for other engines because they use books from human-played games.
Engines don’t win by playing novel moves all the time that humans can’t comprehend. They win by playing the most accurate move every time (never making a mistake).
The analysis in the paper draws the exact opposite conclusion. Alpha zero’s play agrees remarkably with humans despite not seeing a human game (e.g. it found the Berlin defense to the Ruy Lopez).
Also your point is trivially not true for other engines because they use books from human-played games.
Engines don’t win by playing novel moves all the time that humans can’t comprehend. They win by playing the most accurate move every time (never making a mistake).
"Also your point is trivially not true for other engines because they use books from human-played games."
Books are only used in openings, and in (some) endings, in the form endgame tables. Throughout the middle game - where most fireworks happen - an engine is on its own.
And yes, engine's choices are often incomprehensible even for an elite human player (although it isn't literally every second move).
There was this Polish player, Patrycja Waszczuk, accused of cheating (she got banned for 2 years but she appealed, there's an ongoing legal battle).
The case got quite a bit of attention, and parts of her suspicious games ended up analyzed by Nakamura (a world class player, for those unfamiliar with the field).
There've been several moves which left him dumbfounded, as in "no human would even consider that in this position". I don't have a link, but it can be found on YT.
Books are only used in openings, and in (some) endings, in the form endgame tables. Throughout the middle game - where most fireworks happen - an engine is on its own.
And yes, engine's choices are often incomprehensible even for an elite human player (although it isn't literally every second move).
There was this Polish player, Patrycja Waszczuk, accused of cheating (she got banned for 2 years but she appealed, there's an ongoing legal battle).
The case got quite a bit of attention, and parts of her suspicious games ended up analyzed by Nakamura (a world class player, for those unfamiliar with the field).
There've been several moves which left him dumbfounded, as in "no human would even consider that in this position". I don't have a link, but it can be found on YT.
A distinction should be made between coming up with a move and understanding the motivation for the move.
Incomprehensible moves are going to overwhelmingly be from deep searches while fast polynomial time heuristics such as that would come from neural networks are much more likely to be human discernible.
Incomprehensible moves are going to overwhelmingly be from deep searches while fast polynomial time heuristics such as that would come from neural networks are much more likely to be human discernible.
"A distinction should be made between coming up with a move and understanding the motivation for the move."
Sure, but realizing what the reasoning could be in retrospect (sort of reverse engineering) is by definition easier than coming up with a (paradoxical) move in the first place, so if you can't even do the former...
"polynomial time heuristics such as that would come from neural networks are much more likely to be human discernible."
AlphaZero's playing style definitely feels more human-like, as it has a "speculative" quality to it.
To quote no other than Kasparov:
"I admit that I was pleased to see that AlphaZero had a dynamic, open style like my own. The conventional wisdom was that machines would approach perfection with endless dry maneuvering, usually leading to drawn games. But in my observation, AlphaZero prioritizes piece activity over material, preferring positions that to my eye looked risky and aggressive. Programs usually reflect priorities and prejudices of programmers, but because AlphaZero programs itself, I would say that its style reflects the truth."
It's probably a bit deceptive, because what looks risky to us, probably isn't really perceived as "risky" by the neural network, which can "feel" the deeper correctness of the decision, much unlike traditional engines, whose conservative cautiousness stems from their inherent limitations. (We know that now - before AZ it was assumed this is simply the "right" way to play, just like Kasparov says).
Sure, but realizing what the reasoning could be in retrospect (sort of reverse engineering) is by definition easier than coming up with a (paradoxical) move in the first place, so if you can't even do the former...
"polynomial time heuristics such as that would come from neural networks are much more likely to be human discernible."
AlphaZero's playing style definitely feels more human-like, as it has a "speculative" quality to it.
To quote no other than Kasparov:
"I admit that I was pleased to see that AlphaZero had a dynamic, open style like my own. The conventional wisdom was that machines would approach perfection with endless dry maneuvering, usually leading to drawn games. But in my observation, AlphaZero prioritizes piece activity over material, preferring positions that to my eye looked risky and aggressive. Programs usually reflect priorities and prejudices of programmers, but because AlphaZero programs itself, I would say that its style reflects the truth."
It's probably a bit deceptive, because what looks risky to us, probably isn't really perceived as "risky" by the neural network, which can "feel" the deeper correctness of the decision, much unlike traditional engines, whose conservative cautiousness stems from their inherent limitations. (We know that now - before AZ it was assumed this is simply the "right" way to play, just like Kasparov says).
> but realizing what the reasoning could be in retrospect (sort of reverse engineering)
Not in this situation. While the chess engine performed a network guided non-deterministic search and returned a result, its neural network did not. With tools like in the paper, we can now map the network's internal states to something close to human relatable concepts.
Even without such tools we could observe what lines were productive by analyzing a few ply deep and comparing with other moves we could have made. It's closer to something becoming obvious with hindsight, where seeing what works compared to what you thought of helps to highlight the flaw in your plan or something you missed, while reinforcing your understanding of the engine's moves. That scenario is a common enough one.
Not in this situation. While the chess engine performed a network guided non-deterministic search and returned a result, its neural network did not. With tools like in the paper, we can now map the network's internal states to something close to human relatable concepts.
Even without such tools we could observe what lines were productive by analyzing a few ply deep and comparing with other moves we could have made. It's closer to something becoming obvious with hindsight, where seeing what works compared to what you thought of helps to highlight the flaw in your plan or something you missed, while reinforcing your understanding of the engine's moves. That scenario is a common enough one.
I meant "reasoning" as in finding out the objective reason, within the realm of chess itself. Not in how the neural network conducted this reasoning under the hood.
[deleted]
To add to Galanwe's point:
Wikipedia says (about AlphaGo):
"AlphaGo's playing style strongly favours greater probability of winning by fewer points over lesser probability of winning by more points. Its strategy of maximizing its probability of winning is distinct from what human players tend to do which is to maximize territorial gains, and explains some of its odd-looking moves."
In other words optimizing the likelihood of winning looks strange to experts who use other heuristics to assess positions.
Wikipedia says (about AlphaGo):
"AlphaGo's playing style strongly favours greater probability of winning by fewer points over lesser probability of winning by more points. Its strategy of maximizing its probability of winning is distinct from what human players tend to do which is to maximize territorial gains, and explains some of its odd-looking moves."
In other words optimizing the likelihood of winning looks strange to experts who use other heuristics to assess positions.
Especially given the rise of speed chess over the last few years. Focus on solid heuristics and speed, not finding strictly optimal moves.
I'm not a chess player, but I was into competitive Pokémon for a time.
Any decent player will prioritise win probability over loss minimisation, even to the point of sacrificing multiple Pokémon to get a single turn of setup with your main sweeper.
While it seems confusing to newbies, experienced players know exactly why they're making the choices that they are.
Any decent player will prioritise win probability over loss minimisation, even to the point of sacrificing multiple Pokémon to get a single turn of setup with your main sweeper.
While it seems confusing to newbies, experienced players know exactly why they're making the choices that they are.
Right. I’m saying expert human players of deep and complex games still have a heuristic tied to human concepts: piece count, piece value, safety, mobility, etc. So a professional considering a line of play might be turned off that line if one of those measures would suffer.
There are some amazing games of Alpha Zero playing Stockfish (the other best chess program) where it seems Alpha Zero just doesn’t give a fuck. It’s just sacrificing it’s own pieces and not clear (to a human… or Stockfish) what the plan is. Later it achieves a zugzwang against Stockfish, which is the most humiliating thing it could do (if either program had emotions).
There are some amazing games of Alpha Zero playing Stockfish (the other best chess program) where it seems Alpha Zero just doesn’t give a fuck. It’s just sacrificing it’s own pieces and not clear (to a human… or Stockfish) what the plan is. Later it achieves a zugzwang against Stockfish, which is the most humiliating thing it could do (if either program had emotions).
The same is true of Magic: The Gathering where an oft repeated lesson is "life is a resource". Meaning you shouldn't be afraid to spend it to win.
My guess would be that for chess it's more difficult to gauge the difference in real time.
My guess would be that for chess it's more difficult to gauge the difference in real time.
Stockfish prior to adopting a neural network evaluation engine made a lot of questionable-looking moves. But it makes many more natural looking moves after being upgraded to use the NN evaluation engine. Broadly speaking, NN based engines make more human looking moves compared to rule-based evaluations[1], with the human-expert accuracy going up as the human rating increases.
[1]https://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf (scroll down for the graphs)
[1]https://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf (scroll down for the graphs)
I am sorry but you're completely wrong. Modern engine's moves are usually easy to understand if you can play around with the engine running. The engine is just way more precise. Usually the variations needed to understand the moves aren't particularly long either. For me (2300 ELO) it's usually 3-6 ply deep when I go "ahhhh obviously this is better".
The method you're referring to is based on detecting how often a human picks the top move out of 2-3 similarly rated ones and most of the time understanding which one of the 56.5%, 57.1% and 57.3% moves is the best is difficult but understanding why 55% move is better than 51% one is usually pretty easy. I am using AlphaZero/Lc0 evals here btw as I find them more intuitive than traditional engines ones.
I think your opinion might have been formed many years ago. There was indeed a time where engines like traditional Stockfish (before NNUE) were already much stronger than humans but their strategic play made little sense. It turned out that they were in fact weak strategically and modern crop of engines showed it. Try running Lc0 with modern network vs say Stockfish 7 and you will see games which could be described as humiliating defeats one after another.
It turned out that we are actually very good at figuring chess out. Not very precise, slow and inconsistent but we've got most of the high level concepts right.
It doesn't mean engines don't come up with occasional move that is just too deep to get without long analysis and too counterintuitive to consider during play. It's just rare. Most of the moves are intuitive, and make a lot of sense to a human.
The method you're referring to is based on detecting how often a human picks the top move out of 2-3 similarly rated ones and most of the time understanding which one of the 56.5%, 57.1% and 57.3% moves is the best is difficult but understanding why 55% move is better than 51% one is usually pretty easy. I am using AlphaZero/Lc0 evals here btw as I find them more intuitive than traditional engines ones.
I think your opinion might have been formed many years ago. There was indeed a time where engines like traditional Stockfish (before NNUE) were already much stronger than humans but their strategic play made little sense. It turned out that they were in fact weak strategically and modern crop of engines showed it. Try running Lc0 with modern network vs say Stockfish 7 and you will see games which could be described as humiliating defeats one after another.
It turned out that we are actually very good at figuring chess out. Not very precise, slow and inconsistent but we've got most of the high level concepts right.
It doesn't mean engines don't come up with occasional move that is just too deep to get without long analysis and too counterintuitive to consider during play. It's just rare. Most of the moves are intuitive, and make a lot of sense to a human.
Some moves are hard to understand, but it's more like one or two per game. The AI is after all a lot better than Nakamura and Co.
The usual way to detect cheating is detecting incredible accuracy over a whole game, not individual engine like moves.
The usual way to detect cheating is detecting incredible accuracy over a whole game, not individual engine like moves.
You can find out why an engine is making a move though. Hikaru and Magnus won't immediately understand why a move was played but they eventually will as they analyze the game.
> Do you actually play chess? Half the moves made by an AI at 3500+ ELO (e.g. Stockfish 13+) is completely alien to even a human super GM like Nakamura or Carlsen
No they're not. GMs get thrown out on certain moves, because we tend to play chess strategically, that is, we make a plan and try to find ways to make it work. AIs look like they have a plan and then immediately change course when they compute it's beneficial to do so.
As GP says, it takes a minute or two, but those same GMs will (Nakamura does this constantly in some of his streams) reevaluate the strategy, play some lines to see what is tripping the AI and then figure it out. It's that rapid change of strategy from, as an example, extremely defensive to wildly aggressive, which makes them suspicious that they're playing a computer.
> And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding is still leverages the same concepts.
In that I'd agree with you that GP is wrong. For the AI there's no concepts, it's evaluating the chance of success, the perceived "weights" are irrelevant, you could map any number of concepts and get a correlation between them.
No they're not. GMs get thrown out on certain moves, because we tend to play chess strategically, that is, we make a plan and try to find ways to make it work. AIs look like they have a plan and then immediately change course when they compute it's beneficial to do so.
As GP says, it takes a minute or two, but those same GMs will (Nakamura does this constantly in some of his streams) reevaluate the strategy, play some lines to see what is tripping the AI and then figure it out. It's that rapid change of strategy from, as an example, extremely defensive to wildly aggressive, which makes them suspicious that they're playing a computer.
> And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding is still leverages the same concepts.
In that I'd agree with you that GP is wrong. For the AI there's no concepts, it's evaluating the chance of success, the perceived "weights" are irrelevant, you could map any number of concepts and get a correlation between them.
Sure, but when I watch Carlsen or Nakamura okay, their moves often look alien to me.
[deleted]
> Rather, it just does what [less successful people] do - ... - but better and more consistently.
Reminds me of comparisons between Renaissance and other quant funds. If you get enough better it looks like what you're doing is magic until you analyse what was done and see they have better systems and people.
Sometimes, magic is just someone spending more time on something than anyone else might reasonably expect - Teller
Reminds me of comparisons between Renaissance and other quant funds. If you get enough better it looks like what you're doing is magic until you analyse what was done and see they have better systems and people.
Sometimes, magic is just someone spending more time on something than anyone else might reasonably expect - Teller
Shame they could not extend their better people and systems to all their customers:
"Renaissance Clients Exit After Firm’s Anemic Run of Results"
https://www.bloomberg.com/news/articles/2021-02-08/renaissan...
"Renaissance Clients Exit After Firm’s Anemic Run of Results"
https://www.bloomberg.com/news/articles/2021-02-08/renaissan...
After watching some of the games AlphaZero played against Stockfish, I have to say I don't really agree that the moves make sense to a grandmaster... AlphaZero has this uncanny ability to sacrifice many pawns to restrict the movement of the opponent's pieces, something that you might see in advanced play but nowhere near the extent you see it with AlphaZero. Many more of the moves it plays are hard to make sense of compared to a classic engine such as Stockfish.
See GothamChess' very entertaining analysis: https://www.youtube.com/watch?v=8dT6CR9_6l4
See GothamChess' very entertaining analysis: https://www.youtube.com/watch?v=8dT6CR9_6l4
> When a superhuman AI plays chess, the moves usually make sense to an expert.
It's really interesting to look at very high level players analyzing specific AI moves. Many they understand after a short while, others take longer - and some they don't understand directly, but might have an inkling as to why it's good.
My feeling is that many high level analysts would rather just automatically defer to the AI's analysis rather than challenging it - so there is in some sense a lack of feedback to correct such situations. Even in the 1997 Deep Blue vs Kasparov match, game #1 Deep Blue apparently was a bugged position so it moved randomly (a legal random move) - but because Kasparov believed it was correct he felt the analysis was way beyond him so he conceded.
It's really interesting to look at very high level players analyzing specific AI moves. Many they understand after a short while, others take longer - and some they don't understand directly, but might have an inkling as to why it's good.
My feeling is that many high level analysts would rather just automatically defer to the AI's analysis rather than challenging it - so there is in some sense a lack of feedback to correct such situations. Even in the 1997 Deep Blue vs Kasparov match, game #1 Deep Blue apparently was a bugged position so it moved randomly (a legal random move) - but because Kasparov believed it was correct he felt the analysis was way beyond him so he conceded.
But could there be a class of games whose solutions are so complex that we can't understand them, yet an AGI would?
We designed chess to be easily understood by humans. There's a cap on the conceptual complexity that fits within our capabilities. So it's not surprising that we can understand AI solutions that merely asymptote to that fixed cap.
We designed chess to be easily understood by humans. There's a cap on the conceptual complexity that fits within our capabilities. So it's not surprising that we can understand AI solutions that merely asymptote to that fixed cap.
Chess are very limited compared to the whole universe, AGI could for example invent self-replicating fusion-driven paperclip factories, there's nothing in the universe rules that makes this impossible we simply don't know how to do it (yet).
Sure, and God could smite all unbelievers, turn water into wine and bring about the Rapture: there are no limits to what purely hypothetical omniscient beings can do.
But based on we actually know about the sort of AI systems humans actually know how to build and obtain useful results from, we'd end up with a computer which absorbed time series of paperclip sales figures, supply chain costs and marketing campaign data which recommends small paperclips for the Kazakh market, and there's no reason to believe a model trained to optimise paperclip production would know anything at all about how to generate fusion power. This article suggests we might even be able to analyse why the algorithm prioritised that obscure market after applying a lot more thought to the relevant data ourselves. (We can also reason that the AI model is more likely to be wrong about the small paperclips for the Kazakh market, because the permutations for the Kazakh market aren't as perfectly encapsulated in the underlying model and simulated by training on data as the permutations of chess)
But based on we actually know about the sort of AI systems humans actually know how to build and obtain useful results from, we'd end up with a computer which absorbed time series of paperclip sales figures, supply chain costs and marketing campaign data which recommends small paperclips for the Kazakh market, and there's no reason to believe a model trained to optimise paperclip production would know anything at all about how to generate fusion power. This article suggests we might even be able to analyse why the algorithm prioritised that obscure market after applying a lot more thought to the relevant data ourselves. (We can also reason that the AI model is more likely to be wrong about the small paperclips for the Kazakh market, because the permutations for the Kazakh market aren't as perfectly encapsulated in the underlying model and simulated by training on data as the permutations of chess)
The assumption was
> suppose we invent some superhuman AGI for $100B in 2100
not
> AI systems humans actually know how to build and obtain useful results from
You're talking about specialized expert system that is possible today.
BTW: AGI don't need to know anything about fusion power, just like Alpha Zero doesn't know anything about chess when it starts. Its supposed to learn the rules as it goes, and in case of AGI its chessboard is the universe.
Chess are a pretty well balanced and stable environment, so there's no way to cheat, but when you give less well-defined environment to AI it routinely cheats. See for example AlphaStar working around the APM limit by "saving" APM for crucial fights and completely demolishing human players in these fights microing at 1000s actions per minute. Or evolutionary systems learning to walk by abusing numerical instability bugs in the physics engine.
We know there are cheatcodes to reality (see industrial revolution), we know we haven't yet discovered all of them, and the assumption was AGI will be smarter than us. What you're doing is wishful thinking AGI won't find them cause it will be too busy with accounting.
> suppose we invent some superhuman AGI for $100B in 2100
not
> AI systems humans actually know how to build and obtain useful results from
You're talking about specialized expert system that is possible today.
BTW: AGI don't need to know anything about fusion power, just like Alpha Zero doesn't know anything about chess when it starts. Its supposed to learn the rules as it goes, and in case of AGI its chessboard is the universe.
Chess are a pretty well balanced and stable environment, so there's no way to cheat, but when you give less well-defined environment to AI it routinely cheats. See for example AlphaStar working around the APM limit by "saving" APM for crucial fights and completely demolishing human players in these fights microing at 1000s actions per minute. Or evolutionary systems learning to walk by abusing numerical instability bugs in the physics engine.
We know there are cheatcodes to reality (see industrial revolution), we know we haven't yet discovered all of them, and the assumption was AGI will be smarter than us. What you're doing is wishful thinking AGI won't find them cause it will be too busy with accounting.
Sure, but we're also talking about drawing inferences about AGI from how superintelligences based on ML processes actually behave rather than from stories. The current of the state of the art suggests an AGI might not need to know anything about fusion power to invent it, if it is equipped with a system to perfectly simulate the outcomes of experimenting with fusion power like Alpha Zero is with the games it plays. I'm not sure we can accidentally code one of those.
The "cheat" argument suppports my argument, not yours. Rather than do something incredibly difficult and indirect like master advanced physics outside the scope of the simulations it can run to invent new power sources to legitimately produce very low cost paperclips, a system to maximise paperclip sales indicators is probably going to exploit gaps in the simulations it can run or data its already supplied with, like data entry errors in a region or logical holes in the supply chain pricing model, or the marketing simulation not including indications of languages spoken. So it might actually be worse than the management consultants at optimising paperclip revenues. On a related note, if you did want your AGI to invent fusion power but your fusion power emulation engine isn't up to scratch, it's likely to design reactors that are very performant inside the simulation but don't actually work, much like those evolutionary systems exploiting physics engine bugs when faced with the comparatively simple task of learning how to walk.
The "cheat" argument suppports my argument, not yours. Rather than do something incredibly difficult and indirect like master advanced physics outside the scope of the simulations it can run to invent new power sources to legitimately produce very low cost paperclips, a system to maximise paperclip sales indicators is probably going to exploit gaps in the simulations it can run or data its already supplied with, like data entry errors in a region or logical holes in the supply chain pricing model, or the marketing simulation not including indications of languages spoken. So it might actually be worse than the management consultants at optimising paperclip revenues. On a related note, if you did want your AGI to invent fusion power but your fusion power emulation engine isn't up to scratch, it's likely to design reactors that are very performant inside the simulation but don't actually work, much like those evolutionary systems exploiting physics engine bugs when faced with the comparatively simple task of learning how to walk.
There is no evidence to suggest cheat codes in reality. The industrial revolution, your example, was a hard lesson against that. Attempts to cheat thermodynamic laws with perpetual motion machines were summarily shot down.
Or even without using any such new technology, it could at least take advantage of the fact that almost all human communication is mediated by computers, and coerce virtually the entire human race into helping it. I think the GP wasn't being nearly creative enough when thinking about the possibilities/dangers of AGI.
Serious question: I haven't done the research but I've always felt that in abstracts like Chess and Go that a larger part of the skill is in "reading the board". Humans are leveraging our pattern matching but we can get tired or miss even one little thing. The AI never misses, ever. I've always wondered how a human would do against an AI if the human was given a list of the top 50 potential moves in a randomized order to consider before making their move. Would the human be on more level playing field in terms of reading?
There never are 50 reasonable possible moves in Chess, but with top 3 it would be an advantage.
Perhaps it's illustrative to consider the chess variant "hand and brain" played by teams of two people, one ("brain") picking the piece to be moved, and the other ("hand") choosing where to move it; the relationship is complex, but in general having a strong "brain" limit your possible moves to just a single piece would improve the chances of a weaker "hand" above what they could do on their own.
Perhaps it's illustrative to consider the chess variant "hand and brain" played by teams of two people, one ("brain") picking the piece to be moved, and the other ("hand") choosing where to move it; the relationship is complex, but in general having a strong "brain" limit your possible moves to just a single piece would improve the chances of a weaker "hand" above what they could do on their own.
The fallacy here is that we can only measure whether AZ is playing human strategies, by comparing against our own known space of strategies. Sure, we human have developed good heuristics the AI has also found, but we can’t know if it has better/more subtle concepts that we haven’t discovered, exactly because we haven’t discovered them yet. Who is to say “mobility” is even a coherent top level concept compared to some unpronounceable abstraction the AI has figured out?
In the first match of the Lee Sedol / Alpha Go jubango, many moves were initially interpreted as slow, or simply mistakes.
Human players try to maximize score difference, Alpha Go tries to maximize the probability of winning. This means, between playing a risky move that makes more territory, and a safe move that makes less territory, Alpha Go prefers the safer move.
To experts, those move came across as "slow", inefficient or suboptimal. But they turn out to be the superior moves.
Human players try to maximize score difference, Alpha Go tries to maximize the probability of winning. This means, between playing a risky move that makes more territory, and a safe move that makes less territory, Alpha Go prefers the safer move.
To experts, those move came across as "slow", inefficient or suboptimal. But they turn out to be the superior moves.
The difference in strategy makes sense when you consider humans know we are likely to make some variable (suboptimal) choices and so our strategy includes building a buffer to guard against the inevitable gaff.
GAI/NN doesn't 'fear' variability and so it's OK to ride a razor thin margin all the way to the victory.
[edit: rephrase]
GAI/NN doesn't 'fear' variability and so it's OK to ride a razor thin margin all the way to the victory.
[edit: rephrase]
The paper itself is here: https://arxiv.org/abs/2111.09259 with the key conclusion that "Examining the evolution of human concepts using probing showed that many human concepts can be accurately regressed from the AZ network after training, even though AlphaZero has never seen a human game of chess, and there is no objective function promoting human-like play or activations" and "[t]he fact that human concepts can be located even in a superhuman system trained by self-play
broadens the range of systems in which we should expect to find human-understandable concepts"
Aren't the shared concepts simply predicated on 'the rules of chess'? No surprise that an algorithm and a 'rational actor' share similar themes when both are constrained by the same rules.
the concept of material isn't found in the rules and yet was discovered independently during self-play.
> the concept of material isn't found in the rules
It's not spelled out directly, but it is a pretty low-hanging fruit.
After all, the rules dictate that the number of pieces on the board never increases. The rules also dictate the different kinds of moves allowed by each piece. Together it seems to me those rules invariably lead to pieces not having the same value, aka the concept of material.
It's not spelled out directly, but it is a pretty low-hanging fruit.
After all, the rules dictate that the number of pieces on the board never increases. The rules also dictate the different kinds of moves allowed by each piece. Together it seems to me those rules invariably lead to pieces not having the same value, aka the concept of material.
We are quite literally alpha zero ourselves. We developed chess by playing ourselves. I don't get what is particularly surprising that there are not infinite ways to play good chess?
Human minds didn't evolve for playing chess. So there was a possibility that there are "hardware" limitations that prevent us from playng it the way it should be (give or take the inevitable error margin).
It's not a priori obvious that a machine can learn to play chess in a similar way to the way a human can.
I suppose if AGI gets achieved then some of this will seem predictable in hindsight. But we don't yet know if it's achievable.
I suppose if AGI gets achieved then some of this will seem predictable in hindsight. But we don't yet know if it's achievable.
Maybe for more complex games, example: economics. But chess is pretty simple compared to seven billion people going their own ways.
On the other side, AlphaZero and all the previous Deep Mind's AIs played some very different go than human players did. Every single move was not unheard of but they were dismissed by human masters as not good. The most noticeable contribution was the total change of how to play corner invasions. By looking at game records we can draw a line between pre and post AlphaGo. Maybe go is a little bit more wide than chess and AIs have more space to find novel plays.
On the other side, AlphaZero and all the previous Deep Mind's AIs played some very different go than human players did. Every single move was not unheard of but they were dismissed by human masters as not good. The most noticeable contribution was the total change of how to play corner invasions. By looking at game records we can draw a line between pre and post AlphaGo. Maybe go is a little bit more wide than chess and AIs have more space to find novel plays.
We (humans, AlphaZero) are both distilling the rules down. Some rules are easily distilled (a queen is worth more than pawn) some are more difficult (mobility, knight vs bishop). I would expect the easily distilled rules to be discovered by both of us.
What I think would be cool is trying to distill or revise better guidelines from what AlphaZero does.
What I think would be cool is trying to distill or revise better guidelines from what AlphaZero does.
Can this be explained by the fact that the rules of chess evolved accomodating certain preferences that human players developed over the course of time?
Take a much simpler game like paper, rock scissors, where certain strategies exist as well (I hear). This should be much easier to analyze. Can somebody apply alphaZero to rock, paper, scissors, please?
Take a much simpler game like paper, rock scissors, where certain strategies exist as well (I hear). This should be much easier to analyze. Can somebody apply alphaZero to rock, paper, scissors, please?
The human-understandable concepts are there, but I think finding them is still very hard, right? I.e. it doesn’t sound like the system including AZ was able to identify and articulate the human concepts such as king safety a priori.
A game like chess is like two competing mathematical models. The human understanding of it is really just noticing patterns that exist in the model. I would hope an ML algorithm finds similar patterns.
[deleted]
The sci-fi trope that aliens or superintelligent AIs will have thought processes completely different and incomprehensible to humans is probably not true.
If the aliens live in our universe, evolved on a planet with limited resources, need food, breathe, etc., then they'll likely have a lot in common with Earth animals psychologically. If the AI is tasked with doing something that humans also do, then its tactics will be at least recognizable to an expert in that field. Moreso in well-understood systems such as chess.
Is it really plausible that an advanced chess AI's tactics would have no concept of threats or king safety? The same tactics apply to it as do to us.
If the aliens live in our universe, evolved on a planet with limited resources, need food, breathe, etc., then they'll likely have a lot in common with Earth animals psychologically. If the AI is tasked with doing something that humans also do, then its tactics will be at least recognizable to an expert in that field. Moreso in well-understood systems such as chess.
Is it really plausible that an advanced chess AI's tactics would have no concept of threats or king safety? The same tactics apply to it as do to us.
Chess is a closed system. Alien beings could be "playing checkers", so to speak. A biology completely different to anything existing on Earth (say, inteligent life living in the astmophere of a hot gas giant planet) would likely lead an utterly incompatible psychology. It's kind of like in the famous Wittgenstein quote: "if a lion could speak, we could not understand him". Well, lion is an Earth animal, but you see the point.
> then they'll likely have a lot in common with Earth animals psychologically
The term "psychologically" here is interesting because it skips the "biologically" part. What if they don't have neurons like us or AZ? Could one even talk about psychology?
The term "psychologically" here is interesting because it skips the "biologically" part. What if they don't have neurons like us or AZ? Could one even talk about psychology?
Well, that's kind of the point. Sci-fi authors will talk about how Super-AIs or aliens might think completely differently from us, but then impose lots of implicit similarities that would make that implausible.
Sure, non-sapient fungus-like extraterrestrial life-forms will definitely not think like us, even if they have neurons. But "aliens" in most stories implicitly refers to mobile intelligent beings capable of cooperation, communication, developing technology to a level more advanced than our own, and are animal-like and thus need to breathe, eat, drink, maintain homeostasis, etc. Often they'll even have the same body plan as us.
Also, things like the prisoner's dilemma, the tragedy of the commons, and that lying may be beneficial in some circumstances aren't mere artifacts of human psychology, but are straightforward results of game theory.
If a creature is that similar to a human physically, can it really be totally incomprehensible psychologically?
Sure, non-sapient fungus-like extraterrestrial life-forms will definitely not think like us, even if they have neurons. But "aliens" in most stories implicitly refers to mobile intelligent beings capable of cooperation, communication, developing technology to a level more advanced than our own, and are animal-like and thus need to breathe, eat, drink, maintain homeostasis, etc. Often they'll even have the same body plan as us.
Also, things like the prisoner's dilemma, the tragedy of the commons, and that lying may be beneficial in some circumstances aren't mere artifacts of human psychology, but are straightforward results of game theory.
If a creature is that similar to a human physically, can it really be totally incomprehensible psychologically?
I don't want to be too dismissive of the work going into this, but Chess isn't that huge of a search space. Games can quickly end in many scenarios from certain points.
Does having an evaluation system that takes into account king safety only as a consequence of having taken the entire game state into account, mean that it understands king safety? Can you extract these tactics from the AI? Can it explain the moves?
I would be interested in seeing work here that can provide some comprehensibility to the correlations, as that's how we can get to actual novel intuition. "The computer is good with this neural network due to pattern matching" is less actionable than "the computer is good and found this kind of pattern, and we think it relates to this".
EDIT: I think that if anything Chess is an interesting example of humans emulating computers. People sitting around thinking about variations, trying to snuff out what is good/bad. Granted, the fact that you're playing against other humans is pretty important at a higher level, but it's centuries of people sitting down and running through the search space, using our approximations for what are good and bad.
Does having an evaluation system that takes into account king safety only as a consequence of having taken the entire game state into account, mean that it understands king safety? Can you extract these tactics from the AI? Can it explain the moves?
I would be interested in seeing work here that can provide some comprehensibility to the correlations, as that's how we can get to actual novel intuition. "The computer is good with this neural network due to pattern matching" is less actionable than "the computer is good and found this kind of pattern, and we think it relates to this".
EDIT: I think that if anything Chess is an interesting example of humans emulating computers. People sitting around thinking about variations, trying to snuff out what is good/bad. Granted, the fact that you're playing against other humans is pretty important at a higher level, but it's centuries of people sitting down and running through the search space, using our approximations for what are good and bad.
I don't share this view. They would have had a lot in common at a point in time in their history, but who's to say the consequences of millions of years of additional evolution post-energy abundance, let alone the consequences of embryo selection and genetic engineering? We've had a few tens of thousand years of evolution with relative (but still not really) caloric abundance and look how much bigger our brains got. Add nuclear fusion and an extra 50 million years to that nascent process.
I think chess strategy is a flawed way to be thinking about this because we designed the rules and the solution space to be comprehensible to us. It's not surprising that we can understand AI strategy. That simply follows from the solution space itself being small and comprehensible.
I think chess strategy is a flawed way to be thinking about this because we designed the rules and the solution space to be comprehensible to us. It's not surprising that we can understand AI strategy. That simply follows from the solution space itself being small and comprehensible.
When a fly makes first contact, do you care to reason with it in terms it can understand, as to why you are now swatting it out of existence?
Yes, I wave and take a few sloppy shots before I get up in arms. Some take flight (ha!) but some of the more agressive beasts seem to send the message that I should take what's coming for me.
Swarms of agressive insects are also known to be quite defensive and alert the troops.
Take the human out of the loop and consider mating rituals. It amazes me to no end how simple bugs manage to procreate ...
This does of course not meat the common definition of communication, if it involves pheromones and such like. Nevertheless, when another philosopher says that Pain is impossible tondefine, I will show them pain right there.
Swarms of agressive insects are also known to be quite defensive and alert the troops.
Take the human out of the loop and consider mating rituals. It amazes me to no end how simple bugs manage to procreate ...
This does of course not meat the common definition of communication, if it involves pheromones and such like. Nevertheless, when another philosopher says that Pain is impossible tondefine, I will show them pain right there.
"Sugar good. Planet is sugar. Sorry bye". But translated into firing of neurons within the fly brain. Of course there is that concept of other and self that the fly does not have. But this might be more successful with monkeys.
Your other point about motivation to describe their purpose still stands - but the point about there being too limited similarity between conceptions of the world to convey meaning between species - that doesn't work for me. They are situated in the same universe, and it's very likely that there isn't a qualitative difference between their intelligence and our own. Intelligence must be a universal law, something that falls out of the correct interpretation of thermodynamics in open systems.
Interdimensional aliens might have a more varied purpose I suppose, but then their actions wouldn't be intelligible to us at all. They might decide to convert half all life on earth into the opposite chiral form, because reasons. Or rather because there is some unknown mapping between chirality in this plane to the goals within another plain. The chirality of these biological forms within these dimensions being but the lower part of higher dimensional things which are subject to the motivating forces of higher dimensions. Although it's much more likely that any changes would not be coherent at all in our perspective of reality. The purposeful actions of other beings within other perspective frames of totality would more likely look like random thermal/quantum fluctuations within our perspective. There I solved quantum randomness for you.
Interdimensional aliens might have a more varied purpose I suppose, but then their actions wouldn't be intelligible to us at all. They might decide to convert half all life on earth into the opposite chiral form, because reasons. Or rather because there is some unknown mapping between chirality in this plane to the goals within another plain. The chirality of these biological forms within these dimensions being but the lower part of higher dimensional things which are subject to the motivating forces of higher dimensions. Although it's much more likely that any changes would not be coherent at all in our perspective of reality. The purposeful actions of other beings within other perspective frames of totality would more likely look like random thermal/quantum fluctuations within our perspective. There I solved quantum randomness for you.
I think it pretty cool how AlphaZero does not converge to a single best opening move and its variation of opening moves is higher than humans.
Seeing people analyze AlphaZero games on YouTube, it plays a more open and aggressive game of chess and goes for the win instead of the very defensive play of most of the top players today. Some people hope that if people study how AlphaZero plays chess (the best chess player in the world) to improve their game it might make chess more interesting as most games at the high level now end in draws.
Seeing people analyze AlphaZero games on YouTube, it plays a more open and aggressive game of chess and goes for the win instead of the very defensive play of most of the top players today. Some people hope that if people study how AlphaZero plays chess (the best chess player in the world) to improve their game it might make chess more interesting as most games at the high level now end in draws.
It's a somewhat counterintuitive result from game theory that many of the best strategies we know to various non-trivial games are "mixed strategies". This refers to a strategy (presumably) like the one by AlphaZero here, where moves are chosen from a probabilistic distribution, rather than just picking "the best one". The sub-optimal move choice is counterbalanced by the unpredictability.
Mixed strategies in game theory don’t involve making sub-optimal decisions for the sake of unpredictability. At equilibrium every sub-strategy of the mix is a best response (you have multiple best responses).
I think we're differing on unimportant semantics. Consider that optimal mixed strategies don't generally assign equal probability to each choice. If they were all equally good (say, as a pure strategy response) then they shouldn't have different probabilities.
I don't think so. The payoffs depend on the mix of the opponent. You can't say "the probability of sub-strategy x is higher so it must be a better strategy". If it really was a better strategy, you would pick it all the time, not in some mix.
In games with hidden information, yes. In open games? There is an optimal path from start to victory, for one of the players. No need to deviate randomly.
If you always play for the optimal move, give it a thousand games or so, and your opponents will know the optimal game.
They will learn the best opening move from your first move with white, the best reply from your response to it when they play it, white’s best second move from playing that against your best opening move, etc.
Once they know that, playing against them, all games will be the same and they’ll _look_ as good at chess as you.
Especially if the optimal move isn’t far better than the next best move(s) varying your moves can be used to show that isn’t true, and delay the time it takes your opponents to learn that optimal game.
They will learn the best opening move from your first move with white, the best reply from your response to it when they play it, white’s best second move from playing that against your best opening move, etc.
Once they know that, playing against them, all games will be the same and they’ll _look_ as good at chess as you.
Especially if the optimal move isn’t far better than the next best move(s) varying your moves can be used to show that isn’t true, and delay the time it takes your opponents to learn that optimal game.
That may well be true, but it isn't relevant. Game-playing engines are designed to win the position being played, not to maximize winning probability over its playing career.
Stockfish or AlphaZero don't care one bit about what happens in the next game they play.
Stockfish or AlphaZero don't care one bit about what happens in the next game they play.
It isn't necessary either. In chess, as there's no need to worry about interactions between players' information sets, a player can safely apply pure strategies. Winnings approach being maximized over player career by trying to match as close as possible subgame perfect equilibrium from any given start node per game.
Those considerations are not going to be typical. Every perfect information game will have a pure strategy equilibrium.
Why link to Chessbase instead of to the arxiv paper directly? https://arxiv.org/abs/2111.09259
Chessbase is notoriously sketchy and there is a lawsuit against them for blatantly stealing GPL code.
https://stockfishchess.org/blog/2021/our-lawsuit-against-che...
Chessbase is notoriously sketchy and there is a lawsuit against them for blatantly stealing GPL code.
https://stockfishchess.org/blog/2021/our-lawsuit-against-che...
The Chessbase article is a lot more readable than the arxiv paper. Yeah, they did some bad GPL stuff, but that just isn't really relevant to this particular discussion, and the chessbase link is better for the general reader.
Personally, I'd rather not support dishonest, but pretty outlets over direct sources.
Somebody screwed up somewhere, so entire organization must suffer for all eternity?
No? But if I have the opportunity to exercise my preference- I will feel good about doing so, and suggest others do the same.
[deleted]
> Why link to Chessbase instead of to the arxiv paper directly?
Because their article is easier to read, your link isn't even directly to the paper.
And unlike the fake news media who gate-keeps information to ensure their monopoly and to stop you Gell-Mann Amnesia them, Chessbase has included a direct link to the PDF and the abstract at the bottom of their article. That's awesome.
Bike-shedding GPL violations over quality and honest articles doesn't interest lots of people. They can read about the OT stuff in proper HN discussions on it.
Because their article is easier to read, your link isn't even directly to the paper.
And unlike the fake news media who gate-keeps information to ensure their monopoly and to stop you Gell-Mann Amnesia them, Chessbase has included a direct link to the PDF and the abstract at the bottom of their article. That's awesome.
Bike-shedding GPL violations over quality and honest articles doesn't interest lots of people. They can read about the OT stuff in proper HN discussions on it.
[deleted]
>AlphaZero’s neural network evaluation function doesn’t have the same level of structure as Stockfish’s evaluation function: the Stockfish function breaks down a position into a range of concepts (for example king safety, mobility, and material) and combines these concepts to reach an overall evaluation of the position. AlphaZero, on the other hand, outputs a value function ranging from -1 (defeat is certain) to +1 (victory is guaranteed) with no explicitly-stated intermediate steps. Although the neural network evaluation function is computing something, it’s not clear what.
I find it fascinating that we can create something that can "learn" in some sense in a way we don't fully understand. Or am I misunderstanding the meaning of this passage?
I find it fascinating that we can create something that can "learn" in some sense in a way we don't fully understand. Or am I misunderstanding the meaning of this passage?
We do understand how they learn, but the structure of their knowledge is currently hard to discern in the same way that you can't get thoughts or memories out of somebody's head with a MRI scan.
It's interesting to think that if we create a human-level artificial intelligence using ML, we'll probably have no idea how it works.
It's okay, it won't either
To be fair, we have no idea how we think either!
The way in which I think about it is that "learning" (in humans) often occurs is in the form of heuristics - certain rules or shortcuts which are not guaranteed to be optimal, but do a good job at simplifying (the search-state of) the problem. Instead of trying many simple comparisons between legal moves, we instead use a heuristic to score them.
Non-NN chess engines are really all about doing a good job at balancing these heuristics (like it says above). There's fundamentally an upper bound in how good the overall approach can be, simply because the underlying heuristics are defined and selected by humans: there could be many more heuristics that haven't been considered. This version of "learning" is really just removing human forced constraints which were introduced to help learning, by starting from a position without bias.
Non-NN chess engines are really all about doing a good job at balancing these heuristics (like it says above). There's fundamentally an upper bound in how good the overall approach can be, simply because the underlying heuristics are defined and selected by humans: there could be many more heuristics that haven't been considered. This version of "learning" is really just removing human forced constraints which were introduced to help learning, by starting from a position without bias.
Thanks so much for the response! It's really interesting stuff.
What I want to know is if it is possible to do the opposite: extract novel, useful and human-understandable heuristics from a chess engine?
>From recorded data, we can see that everyone seemed to play e4 in the 1500s. Over centuries, moves like d4, Nf3 or c4 emerged as credible and fashionable alternatives.
The graph they have seems to suggest the opposite - nobody playing e4, just d4 early on (and AZ also gravitating most towards d4).
The graph they have seems to suggest the opposite - nobody playing e4, just d4 early on (and AZ also gravitating most towards d4).
To me, this is a great illustration that superhuman intelligence is not magic. When a superhuman AI plays chess, the moves usually make sense to an expert. And even if they don't immediately make sense, they usually make sense once the expert plays out some lines to see what happens. Superhuman AIs have crushed humans at chess not by discovering new, wildly counterintuitive openings that dumb humans missed (it actually plays many of the same openings). Rather, it just does what humans do - win pieces, keep your king safe, attack, gain space, restrict your opponent's moves - but better and more consistently.
This paper builds on that concept and finds that not only are a superhuman AI's moves usually understandable by experts, but even some of the superhuman AI's internal representations are understandable by experts (!).
In the 5th century BCE, Greek philosophers thought the Earth was a sphere. Although they were eventually improved upon by Newton's ellipsoid, they still had arrived at the right concept. And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding still builds upon the same concepts.
Thinking broadly, like, suppose we invent some superhuman AGI for $100B in 2100. And then we ask it how to optimize our paperclip production. I don't think its recommendations will be magic. Paperclip production is a fairly well understood problem: you bring in some metal, you reshape it, you send it out. It's probably gonna have much of the same advice we've already figured out: invest in good equipment, take advantage of economies of scale, choose cheap but effective metals, put factories near sources of energy and labor and raw materials, ship out paperclips in a hierarchical network with caches along the way to supply unanticipated demand, etc. Like, an AGI's suggestions may well be better than a management consultant's, but it's not like it's going to wave a magic wand and flip the laws of physics. Rather, I expect it to win within the same frameworks we've developed, but by doing things better and more consistently than humans.