Google has a separate user-agent identifier in for robots.txt for AI products that can be blocked separately from the search crawler. Many sites block the AI crawler while allowing the google search crawler. AI content is controlled using googlebot-extended user-agent
I think this problem derives from the nature of and incentives of people. What happens is that certain managers (and founders) have derived their success from pushing certain directions that have added a lot of value.
Usually long term initiative take a lot of domain knowledge and a specific set of connections. When something new becomes more important, they risk losing their personal position since they may not be the best person in terms of knowledge and connection to carry out the new direction. Instead they try to use their accumulated power to keep their direction supported even when it's not in the interest of the business for as long as they can.
To address this, perhaps there's merit in rewarding timely exits and somehow punishing people that have dragged things out. However the most that companies can do is usually just fire someone. And at that point they have probably sucked the company dry of the value they can personally extract.
I wonder if taking such a view can be short-sighted. While it is true that employee turn-over can be costly, perhaps that is also one of the ingredients that make tech successful. The valuable employees that you hire have gained their skills at a number of other companies and the employees that have left are contributing to other companies. While this isn't on your balance sheet, it is contributing to the tech community as well. I wonder how you would go about quantifying this.
From my experiences, those in tech for the most part have come from the tech community and stayed in the tech community. Employee mobility is creates a mixture of ideas and also lets individuals to be trained in certain disciplines by those that are most capable at it. In return, companies get employees that were well trained by other companies thus increasing the value of tech companies as a whole. I wonder if this becomes a trend and companies try to close their borders tighter (somewhat reminiscent of U.S. politics right now, huh?) that companies will see short term gains, while the entire industry would suffer in the long run.
As you get experienced, I think it's beneficial to be able to learn to be both fast/sloppy and slow/meticulous given the circumstances. Having worked in both enterprise scenarios and startup environments, I've learned that it's really a result of larger environmental pressures.
If you're working in long-range software, it's by far more beneficial to be a bit slower and more meticulous in your approach. What you're really looking for is incremental improvements that have large impact because the scale or importance of the project is already established to be very big. While you feel like you're not getting much done, the impact multiplier in these situations is very large.
However, when you're in a startup environment, you don't have any value in the product yet, so you have to move fast. A few bugs here and there are not really a problem since you will probably redesign and revamp different features. Testing is a lower priority and you feel like you're moving a lot faster as a developer.
I think healthy teams have a healthy spread on these trade-offs, centered about around the maturity of the team/product/company. Tension between the two sides keeps things moving forward and allows adjustments to keep the machine moving forward as best it can.
I actually think this is exactly the point. He is talking about how these people are highly intelligent but not successful because society isolates them and doesn't give them the opportunity and motivation to learn those skills in the appropriate window.
This is kind of the opposite of students that are underperforming that don't get the attention to help them keep up. Instead these are students that can overperform, but don't get the challenges, validation, and motivation - the general positive feedback - that will help them grow and succeed. Instead they are meet with disapproval and neglect since teachers both dislike them asking hard questions (thus challenging their authority among already hard to control kids) and don't have time to give them the attention they need (since they are most focused on the largest median group).
See https://developers.google.com/search/docs/crawling-indexing/...
In fact NYT does block google from using its content for GenAI products:
User-agent: Google-Extended Disallow: /
As well as chat gpt
User-agent: GPTBot Disallow: /
So idk why this would be unexpected.
See https://www.nytimes.com/robots.txt