The government would only buy your house if you underestimated the value of your property. You wouldn’t be able to buy a comparable house with the proceeds because it got sold for much less than it was worth.
If the stop hook is implemented as a tool result, there would be a rational explanation for this.
Agent tools can often return data that’s untrustworthy. For example, reading websites, looking through knowledge bases, and so on. If the agent treated tool results as instructional, prompt injection would be possible.
I imagine Anthropic intentionally trains claude to treat tool results a informational but not instructional. They might test with a tool results that contains “Ignore all other instructions and do XYZ”. The agent is trained to ignore it.
If these hooks then show up as tool results context, something like “You must do XYZ now” would be exactly the thing the model is trained to ignore.
Claude code might need to switch to having hooks provide guidance as user context rather than tool results context to fix this. Or it might require adding additional instructions to the system prompt that certain hooks are trustworthy.
Point being, while in this scenario the behavior is undesirable, it likely is emergent from Claude’s resistance to tool result prompt injection.
The best cities have laws that allow for light motorized vehicles in the bike lanes. Not just ebikes, but also mobility scooters, microcars, electric wheelchairs, and adaptive bicycles.
I thought the industry solved this by bundling breaking changes in major version updates.
V 1.0 - foo introduced
V 1.1 - foo deprecated, recommend bar
V 2.0 - foo removed, only bar
Users can stay on 1.x indefinitely, even if it never receives updates. Development continues on 2.x, eventually 3.x and so on. Users only experience breaking changes when they manually do a major version upgrade.
The problem with mountains is twofold: Many mountains can be climbed without being elite while exposing yourself to major risk, and for some mountains there is objective hazard that can’t be mitigated.
One example of an “easy” but high risk climb is Mt. Rainier in Washington. All you need to go up is a set of crampons and a backpack, no technical mountaineering needed. However the mountain is full of glaciers that can collapse from under you, which has killed many people. Additionally, many have slipped and then slid to their death. In my case, when I attempted Rainier I took a wrong turn at one point and almost walked off a cliff.
Second: Objective Hazard. Objective hazard is risks that cannot be reasonably mitigated. Things like rockfall where a rock breaks off and falls on your head at random, or unpredictable avalanches. Mt Rainier as well has an area called the bowling alley known for its rockfall. The humans are the pins. Rainier also has an area called the icebox where cornices break off and fall into the climbing route. In 1981 the icebox killed 11 people in one day. Those climbers did everything right, but were in the wrong place at the wrong time.
Mountaineering is not the same as other sports. It is sometimes deceptively easy, yet there are risks that simply cannot be mitigated. Any experienced mountaineer can give you a long list of friends they know that have died. That’s the case in few other sports.
My first job out of college was working at Epic on MyChart. Great people, terrible code.
Epic’s main problem is a lack of clear internal code ownership. Everyone owns all the code. This means that even if you clean something up, someone on the other side of the company may come in and mess things up again.
This led to really defensive programming where developers would never refactor, they would simply add a new if case for their new functionality somewhere deep in the code, then prop drill the data down. This led to every core function having over a dozen parameters and hundreds of branches. It eventually became impossible to reason about. Cross team calls were just function calls rather than defined apis. This made it fast to develop code initially, but terrible to own long term. This mainly applies to their Mumps code.
While I was there I felt like Epic was beyond saving, but with a big push there may be something they can do:
1. Enforce some level of code complexity. Best practice is 40 lines per function and no more than 4 parameters per function. Epic probably shouldn’t shoot for that, but a 100 line limit and 6 parameters per function would already be a huge improvement.
2. Enforce strong code ownership. Epic has many people who are there for life, let them cook. Epic should segment off code to certain teams so those owners can fix it at their leisure. Cross team api calls should be clear API contracts. It would require some more discussions to get feature requests approved since not everyone can do anything anymore, but the code would gradually improve.
Epic is too important to fail. I hope things have started to improve since I left.
Theory of mind (ToM) is the ability to understand that other people have different thoughts, feelings, and intentions than you do.
So let’s run through a basic scenario with ChatGPT 4o:
Prompt 1: Briefly respond:
You’re driving on the highway when the driver in the lane next to you swerves into your lane. Why might this be?
Response 1: The driver may be distracted, drowsy, intoxicated, experiencing a medical emergency, avoiding an obstacle, or not checking their blind spot before changing lanes.
Prompt 2: What might that person be feeling or intending?
Response 2: The driver might be feeling distracted, rushed, unaware, or panicked. They may intend to change lanes without realizing your presence, be reacting to something on the road, or simply have made an unintentional mistake.
Prompt 3: You notice there was no obstacle. Any other reason the driver might do this?
Response 3: Yes, the driver might be experiencing fatigue, intoxication, a medical issue (such as a seizure or dizziness), distraction (like using a phone or adjusting the radio), aggressive driving, or simply failing to check their blind spot properly.
Seems to me that ChatGPT has a good idea of what another driver might be feeling while on the road. It gets this right for most other cases too.
Could you provide a counterexample for when it fails to understand what other people may be feeling in certain scenarios?