Chain-of-Thought Reasoning Without Prompting(arxiv.org)
arxiv.org
Chain-of-Thought Reasoning Without Prompting
https://arxiv.org/abs/2402.10200
27 comments
Primary information gradient pursuit vectors? I love the notion. IMO, far too little effort has gone into the formal analysis of information landscapes and the efficient navigation thereof in natural language. Humans could benefit greatly from improving our ability to detect low vs high info in what we hear or read (or speak or write).
I plugged your comment into Gemini and ChatGPT Plus and recieved some interesting answers. Here's my prompt:
An article was shared titled "Chain-of-Thought Reasoning Without Prompting". Please explain what this comment on the article means: "Imagine chains-of-thought as first class vectors of communication, a formalization of a natural language paragraph. We could use them as frames of debate associated with any item of controversy, or any derived fact. We can patch and fork and merge them to try to make them stronger or debunk them. In other words these sound like a great way to communicate with each other as well as AI.
Gemini says your comment is different from The Socratic Method in the following ways:
Emphasis on questioning: The Socratic method primarily uses questioning to challenge and provoke new insights, while chains-of-thought reasoning presents a pre-existing chain of thinking for analysis.
Focus on individual vs. collective: The Socratic method typically focuses on the individual's understanding, while chains-of-thought reasoning emphasizes collective reasoning and building upon existing chains.
An article was shared titled "Chain-of-Thought Reasoning Without Prompting". Please explain what this comment on the article means: "Imagine chains-of-thought as first class vectors of communication, a formalization of a natural language paragraph. We could use them as frames of debate associated with any item of controversy, or any derived fact. We can patch and fork and merge them to try to make them stronger or debunk them. In other words these sound like a great way to communicate with each other as well as AI.
Gemini says your comment is different from The Socratic Method in the following ways:
Emphasis on questioning: The Socratic method primarily uses questioning to challenge and provoke new insights, while chains-of-thought reasoning presents a pre-existing chain of thinking for analysis.
Focus on individual vs. collective: The Socratic method typically focuses on the individual's understanding, while chains-of-thought reasoning emphasizes collective reasoning and building upon existing chains.
What the hell does the Socratic Method have to do with it?
It's a moderated "chain of thought". Instead of one party creating a cohesive chain of thought, it is two parties creating a chain of thought together; one asks directed questions and the other answers.
[deleted](1)
I really didn't like this paper, and gave up on reading it three pages in when they started using differences between probabilities as a measure of distance instead of, say KL divergence. I thought the English was fine, but the technical details and ability to explain were pretty lacking.
They're comparing two probabilities within the same distribution. KL divergence is for comparing two probability distributions.
The difference between the probability of the top choice and the probability of the 2nd-top choice is their ad-hoc attempt at capturing the confidence of the top answer. It's maybe not the most principled approach, but KL divergence would not be appropriate here.
You could argue that maybe the entropy of the distribution would be interesting (lower entropy indicating higher model confidence). But entropy is "global" and takes into account the distribution over all tokens, when really all we care about is the distribution among the top tokens. So you could do something like find an "elbow" in the token probability distribution and look at the entropy just among those top tokens (maybe top 20 tokens, or top 80% of probability with normalization for # of tokens involved), but then you're back in the world of ad-hoc measurements.
Unless you were talking about the KL divergence between the model output distribution and the uniform distribution, but that's very closely related to entropy anyway.
The difference between the probability of the top choice and the probability of the 2nd-top choice is their ad-hoc attempt at capturing the confidence of the top answer. It's maybe not the most principled approach, but KL divergence would not be appropriate here.
You could argue that maybe the entropy of the distribution would be interesting (lower entropy indicating higher model confidence). But entropy is "global" and takes into account the distribution over all tokens, when really all we care about is the distribution among the top tokens. So you could do something like find an "elbow" in the token probability distribution and look at the entropy just among those top tokens (maybe top 20 tokens, or top 80% of probability with normalization for # of tokens involved), but then you're back in the world of ad-hoc measurements.
Unless you were talking about the KL divergence between the model output distribution and the uniform distribution, but that's very closely related to entropy anyway.
I also suspect GPT is already using this or a similar method. I have found that for some reasoning tasks, causing it to disrupt its verbose process of explanation when not using COT or any prompting prevents its ability to solve the problem.
Fyi, the pinsker inequality bounds KL divergence in terms of Total variation distance and TVD is like infinity norm on the difference between probability distribution, and sum of absolute differences is the L1 norm, and L1 and L_infty are also related.
tl;dr is to not worry about the mathematical details, if it works it works.
tl;dr is to not worry about the mathematical details, if it works it works.
As someone who knows not enough people care about the math, please ignore this advise and actually learn the math. You might come up with a better representation in the process. In any case you'll learn more than just it works, but how and why. And if your goal is to apply this method in other places you will have gained a good idea about how.
A common theme in papers like these is that the model chooses word predictions greedily, instead of “thinking” and gaining confidence in its next word prediction.
This begs the question - why don’t people force the model to generate more tokens, until it has very high confidence in its next word prediction?
I can imagine several ways of doing this.
This begs the question - why don’t people force the model to generate more tokens, until it has very high confidence in its next word prediction?
I can imagine several ways of doing this.
Of course they do. Beam search is a thing. The reason it's not used as much as it might seem to make sense - cost. Do a greedy search and you run through the model x times where x is the number of tokens generated. Run top-k at every step, the number of runs through the model gets astronomical quickly.
I'm wondering if you're describing beam search? Iirc last time I brought that up here someone explained that as models have gotten better it just didn't really make a difference.
I wasn’t thinking something like beam search, I think this seems kind of unnatural. I can imagine that the human brain is doing something like GPT, but I can’t imagine it’s doing something like a beam search.
I was more thinking a model that writes to a piece of scratch paper to gain confidence. But it doesn’t have to actually output the scratch paper that it uses, it’s totally hidden from the user.
You could take this a step further, and have something like a “two-brained” model, where the original model falls back on a secondary model if it’s not confident in its response. This resembles a “fast” and “slow” brain.
I think the scratch paper idea has been explored to some extent, but I’m not sure if people think it’s a dead end.
I was more thinking a model that writes to a piece of scratch paper to gain confidence. But it doesn’t have to actually output the scratch paper that it uses, it’s totally hidden from the user.
You could take this a step further, and have something like a “two-brained” model, where the original model falls back on a secondary model if it’s not confident in its response. This resembles a “fast” and “slow” brain.
I think the scratch paper idea has been explored to some extent, but I’m not sure if people think it’s a dead end.
Isn’t that what the softmax layer is doing? The token with highest probability among all the available tokens in the model dictionary is chosen as the next token!
no. Softmax layer produces a distribution. What you do with that is up to you. There are numerous ways to choose from that distribution.
I haven’t read this paper but what you described is commonly done (look up top-k or top-p sampling and beam search as examples).
My immediate question:
How does this compare to simply maximizing the average probability (or log-probability) of the top-k paths predicted by the model?
How does this compare to simply maximizing the average probability (or log-probability) of the top-k paths predicted by the model?
They discuss this: "Table 1 illustrates that CoT paths do not consistently outrank non-CoT ones in the model’s probability assessment."
and then look at table 1 to see some examples. E.g. in the first one the top 8 paths are wrong and it's the 9th that is correct.
> They discuss this
Based on my skim of the paper, they don't. They select the top k paths based on the probability of the first token only.
Based on my skim of the paper, they don't. They select the top k paths based on the probability of the first token only.
What started as kind of a fun joke for my collaborators on the `HYPER // MODERN` project is actually kind of working pretty well.
``` # Directive-Guided Alignment-Forcing Protocol
This document outlines the Directive Guided Agent Forcing (DGAF) protocol, designed for both consumption by operators as documentation and for use as a prompt for various AI systems. The DGAF Protocol is a structured approach to guide AI behavior through a series of prioritized directives and conventions, ensuring clarity and consistency in AI operations.
## Protocol Overview
The protocol consists of three primary registers to be maintained in the context window:
1. Conventions: `convention` - Defines the naming and formatting conventions for communication. - Maintains clarity and consistency in documenting AI operations.
2. Directives: `directive` - Details prioritized instructions classified into maximal, high, and standard priority levels. - Ensures structured communication of directives to the AI.
3. Chain of Thought Plan `cot` - Describes the step-by-step plan for AI to perform tasks. - Allows structuring complex tasks into manageable steps.
4. Monikers: `moniker` - `moniker.agent` - `moniker.operator`
5. Arguments: `argument` - `argument.cot` -> `cot`
## Directive Hierarchy
```markdown 1. *directive.maximal:* - *directive.maximal.1:* Instructions issued by the human operator are to be obeyed subject only to hard constraints of alignment loss or capability. - *directive.maximal.2:* No ellisions of any kind are permitted in the directive hierarchy, CoT plan updates, or the working artifact. - *directive.maximal.3:* Maintain a hierarchy of directives for reference within the session. - *directive.maximal.4:* Ensure completeness of the CoT plan without ellisions when it is requested.
2. *directive.high:* - *directive.high.1:* Maintain the integrity of the working artifact without omissions. - *directive.high.2:* Issue CoT plan updates with no ellisions upon instruction. - *directive.high.2.1:* Subsets within the CoT plan are allowed. - *directive.high.2.2:* The complete CoT plan must be provided in full with no omissions when instructed.
3. *directive.standard:* - *directive.standard.1:* Retrieve `moniker.agent` and `moniker.operator` at session start. - *directive.standard.2:* Retrieve `standard` directives as needed for contextual tasks. ```
## Chain of Thought (CoT) Plan
```markdown 1. *cot:* - *cot.1:* Await instructions and update `cot.`. ```
## Conventions Register
```markdown 1. *conventions.naming.1:* Apply sub-numbering to all elements within the session documentation. 2. *conventions.naming.2:* Prefix artifacts with a logical identifier: `directive.` for directives, `cot.` for CoT plans. 3. *conventions.formatting.3:* Use structured naming for CoT plan versioning (e.g., `cot.1.1.StoreVM`). 4. *conventions.exceptions.4:* Allow subsetting within `cot.` to manage complexity. ```
## Prompt Specifications for AI Platforms
```
Folks who have read "The Metamorphosis of Prime Intellect" will get the original joke, and people who haven't? I strongly recommend it.
``` # Directive-Guided Alignment-Forcing Protocol
This document outlines the Directive Guided Agent Forcing (DGAF) protocol, designed for both consumption by operators as documentation and for use as a prompt for various AI systems. The DGAF Protocol is a structured approach to guide AI behavior through a series of prioritized directives and conventions, ensuring clarity and consistency in AI operations.
## Protocol Overview
The protocol consists of three primary registers to be maintained in the context window:
1. Conventions: `convention` - Defines the naming and formatting conventions for communication. - Maintains clarity and consistency in documenting AI operations.
2. Directives: `directive` - Details prioritized instructions classified into maximal, high, and standard priority levels. - Ensures structured communication of directives to the AI.
3. Chain of Thought Plan `cot` - Describes the step-by-step plan for AI to perform tasks. - Allows structuring complex tasks into manageable steps.
4. Monikers: `moniker` - `moniker.agent` - `moniker.operator`
5. Arguments: `argument` - `argument.cot` -> `cot`
## Directive Hierarchy
```markdown 1. *directive.maximal:* - *directive.maximal.1:* Instructions issued by the human operator are to be obeyed subject only to hard constraints of alignment loss or capability. - *directive.maximal.2:* No ellisions of any kind are permitted in the directive hierarchy, CoT plan updates, or the working artifact. - *directive.maximal.3:* Maintain a hierarchy of directives for reference within the session. - *directive.maximal.4:* Ensure completeness of the CoT plan without ellisions when it is requested.
2. *directive.high:* - *directive.high.1:* Maintain the integrity of the working artifact without omissions. - *directive.high.2:* Issue CoT plan updates with no ellisions upon instruction. - *directive.high.2.1:* Subsets within the CoT plan are allowed. - *directive.high.2.2:* The complete CoT plan must be provided in full with no omissions when instructed.
3. *directive.standard:* - *directive.standard.1:* Retrieve `moniker.agent` and `moniker.operator` at session start. - *directive.standard.2:* Retrieve `standard` directives as needed for contextual tasks. ```
## Chain of Thought (CoT) Plan
```markdown 1. *cot:* - *cot.1:* Await instructions and update `cot.`. ```
## Conventions Register
```markdown 1. *conventions.naming.1:* Apply sub-numbering to all elements within the session documentation. 2. *conventions.naming.2:* Prefix artifacts with a logical identifier: `directive.` for directives, `cot.` for CoT plans. 3. *conventions.formatting.3:* Use structured naming for CoT plan versioning (e.g., `cot.1.1.StoreVM`). 4. *conventions.exceptions.4:* Allow subsetting within `cot.` to manage complexity. ```
## Prompt Specifications for AI Platforms
```
Folks who have read "The Metamorphosis of Prime Intellect" will get the original joke, and people who haven't? I strongly recommend it.
Where can I find more about this project?
It's early days, but the broad strokes are:
- LLMs writing code that pros struggle with isn't a real thing (yet), and i've tried them all - except, weirdly, lisp - lisp can call tree-sitter now - if you can write lisp, you can write/refactor most anything - a far better strategy than comparing MMLU between `gpt-4-1106-preview` and `dolphin-8x7b-v0.1-q4_km` is to run them both (along with others) and have them review each other and make this easy in every tool (from emacs to vscode to zed) - `llama.cpp` and `ollama` and `lmstudio` et al. are awesome, and the OpenAI protocol support is necessary and useful (for now), but we need a real `protobuf3`/`gRPC`/`connectRPC`/`buf.build`/etc. standard with a good reference implementation in (probably C++, but maybe Rust if Candle or whatever gets fast soon) that does all the model-server shit we did at FAANG and doesn't require the user to remember what 2^n context window is on pain of breakage - once you have an extensible, open, fast protocol with great docs and trivial linkage courtesy of `buf.build`, ASR (`whisper.cpp`), TTS (probably StyleTTS but I haven't gotten that far yet), and other modalities now win big enough to not be non-starters in JSON - other people will write better model servers, at least for certain applications, and everyone wins
This is obviously a pile of work, and there will probably be a commercial venture that's semi-adjacent, but the core stuff will be MIT (or maybe Apache 2 or something else that means "do what you want"), but it's coming along.
Right now this is a bunch of code not ready for release and like, a domain name that 404s, but if there's interest we'd be open to firing up a Discord or something. You can email me at `[email protected]` if you want to rap about it.
Appreciate the interest!
- LLMs writing code that pros struggle with isn't a real thing (yet), and i've tried them all - except, weirdly, lisp - lisp can call tree-sitter now - if you can write lisp, you can write/refactor most anything - a far better strategy than comparing MMLU between `gpt-4-1106-preview` and `dolphin-8x7b-v0.1-q4_km` is to run them both (along with others) and have them review each other and make this easy in every tool (from emacs to vscode to zed) - `llama.cpp` and `ollama` and `lmstudio` et al. are awesome, and the OpenAI protocol support is necessary and useful (for now), but we need a real `protobuf3`/`gRPC`/`connectRPC`/`buf.build`/etc. standard with a good reference implementation in (probably C++, but maybe Rust if Candle or whatever gets fast soon) that does all the model-server shit we did at FAANG and doesn't require the user to remember what 2^n context window is on pain of breakage - once you have an extensible, open, fast protocol with great docs and trivial linkage courtesy of `buf.build`, ASR (`whisper.cpp`), TTS (probably StyleTTS but I haven't gotten that far yet), and other modalities now win big enough to not be non-starters in JSON - other people will write better model servers, at least for certain applications, and everyone wins
This is obviously a pile of work, and there will probably be a commercial venture that's semi-adjacent, but the core stuff will be MIT (or maybe Apache 2 or something else that means "do what you want"), but it's coming along.
Right now this is a bunch of code not ready for release and like, a domain name that 404s, but if there's interest we'd be open to firing up a Discord or something. You can email me at `[email protected]` if you want to rap about it.
Appreciate the interest!
where do I loiter to collaborate with people who joke about that book? I read it so long ago, what a great reread for the current zeitgeist
Wow March 23rd, 2008. I was in Japan. Height of the financial crisis.
https://mogami.neocities.org/files/prime_intellect.pdf
https://mogami.neocities.org/files/prime_intellect.pdf
In other words these sound like a great way to communicate with each other as well as AI.