AI doesn't know how to interact with touchscreens(blog.allada.com)
blog.allada.com
AI doesn't know how to interact with touchscreens
https://blog.allada.com/give-an-llm-an-api-and-itll-thrive-give-it-a-touchscreen-and-it-struggles/
2 comments
I think the big thing I was trying to highlight in this article was the fact the not much effort has been put into spatial and image awareness. In my limited experiments where I would manually ask the models to take an image and highlight things (like "circle all elbows") it does a great job... but if you ask the model where an elbow is in the image (in pixels), it does a poor job.
Or maybe put another way, going from `image->model->tool` seems to be an area for improvement.
Or maybe put another way, going from `image->model->tool` seems to be an area for improvement.
I mean if we see the cost of training then making more apis for everything we have makes sense to me.
what do u think?