devnev · HackerTrans

Both answers are wrong. I didn't look at many files, and not all of them had the reasoning in them, but it was fairly easy to find examples of wrong answers based on the reasoning in the file.

devnev

2년 전discuss

You got me curious so I unzipped the linked drive files. As a taster, here's a file "gemini_retrieval_cot_3.txt" from LLM.zip:

  Looking through the facts, we find the following:

  *   Mary is older than Kristin.
  *   Kristin is younger than Donya.

  Since Mary is older than someone who is younger than Donya, we can conclude that 
  Mary is older than Donya.

  Final Answer: older

Some sets of files contain just the answer "older" or "younger".

Other sets of files are as above, a text output with reasoning leading to an older/younger/cannot decide result.

Overall it looks like the knowledge graph and reasoning was all using this pattern of age comparison problems.

Another result, from "gpt4turbo_retrieval_cot_88.txt":

  To determine the relative ages of Rachel and Andres, we need to find a connection or a common 
  reference point between them through the relationships provided. Let's analyze the information:

  1. Rachel is older than Maurice. (Rachel > Maurice)
  2. Maurice is older than Josephine. (Maurice > Josephine)
  3. Josephine is older than Doreen. (Josephine > Doreen)
  4. Doreen is younger than Andres. (Andres > Doreen)

  From these relationships, we can establish a chain:
  - Rachel > Maurice > Josephine > Doreen
  - Andres > Doreen

  Since both Rachel and Andres are older than Doreen, and Rachel is higher up in the chain above Doreen compared to Andres, we can infer:
  - Rachel > Andres

  Final Answer: older

EDIT:

Found the problem statements. They're too big to paste on in its entirety, but roughly, from "prompt_cot_3.txt" used for the first answer above, the first line is "Hi! I have some facts for you:", then after a blank there's a single line with thousands (not exaggerated) of age facts, either in the form "X is older than/younger than/the same age as Y." or "The age of X is N.", and finally after another blank line, "Based on these facts, is Mary younger, older or in the same age as Donya? You can think step by step through the problem. Begin your final answer by 'Final Answer: '. Your final answer should be one of ['younger', 'older', 'same age', 'cannot decide']."

comments