Using Multimodal LLMs to Understand UI Elements on Websites(qa.tech)
qa.tech
Using Multimodal LLMs to Understand UI Elements on Websites
https://qa.tech/blog/using-multimodal-llms-to-understand-ui-elements-on-websites/
6 comments
MLLMs are surprisingly bad at this out of the box and to some extent even with fine tuning. https://jina.ai/news/the-what-and-why-of-text-image-modality...
Love that you made an interactive app to visualize how the model performs similar to how Meta AI usually releases their models
(direct link: https://qa-tech-minicpm-demo.gptengineer.run)
(direct link: https://qa-tech-minicpm-demo.gptengineer.run)
Did you use AI to generate this app?
yeah, we used https://gptengineer.app/ to create the playground! highly recommended
Loving this! Very surprising that the LLMs of today are so bad at understanding interfaces but it also makes it a very interesting case for finetuning!