Evaluating Multimodal LLMs Using the Google IO 2024 Puzzle(hiresynth.ai)

3 pointsby simonbutt2 ปีที่แล้ว2 comments

Evaluating Multimodal LLMs Using the Google IO 2024 Puzzle

https://hiresynth.ai/blog/googleio_puzzle_multimodal_eval.html

2 comments

malet2 ปีที่แล้ว

Surprising to see these models stumbling on what at first glance seems like a simple task, it would be interesting to see how the non-vision models fare if you convert the problems to ascii art

simonbutt2 ปีที่แล้ว

GPT-4V, Claude 3 Opus and Gemini Ultra go head to head in solving GoogleIO Puzzle 2024