Apple researchers develop AI that can 'see' and understand screen context(venturebeat.com)
venturebeat.com
Apple researchers develop AI that can 'see' and understand screen context
https://venturebeat.com/ai/apple-researchers-develop-ai-that-can-see-and-understand-screen-context/
6 comments
We (Google Research) have a similar work that we recently published: https://blog.research.google/2024/03/screenai-visual-languag...
jwz has a rant (not linked here because he presents a special page for hacker-news referral links) about people sending him screenshots of errors instead of typing the error text into an email. I kind of understood where he was coming from (why encode text as an image, that requires more congnitive load to decode than the original text) but I actually frequently see error screenshots that contain more contextual detail (the OS, what extra software is on the machine, etc) that can help in understanding a bug.
To me, having systems that can do developer-level triage from screen contexts would be pretty valuable although in principle, it shouldn't be needed.
To me, having systems that can do developer-level triage from screen contexts would be pretty valuable although in principle, it shouldn't be needed.
This is interesting in the context of the Rabbit R1-style AI. I forget what they called it, but it’s learning how to ‘drive’ your computer like you do vs. interacting with an API.
So to book a flight it just pretends to be a human and uses the website.
I see a fascinating future where you can teach an AI how to pay your credit card bill. Because I can explain it to you trivially:
1. Log in to credit card provider website.
2. Look at {this number} on the screen. That’s what I owe this month.
3. Log in to my bank.
4. Click Pay > choose ‘credit card’ from my ‘stored contacts’ > paste that amount.
5. Wait for 2FA code to arrive via SMS.
6. Paste 2FA code.
Yes please. I’d like to be able to script my life like this, such that the AI can adapt when my credit card provider changes their CSS class names or whatever would break my fragile Node script.
So to book a flight it just pretends to be a human and uses the website.
I see a fascinating future where you can teach an AI how to pay your credit card bill. Because I can explain it to you trivially:
1. Log in to credit card provider website.
2. Look at {this number} on the screen. That’s what I owe this month.
3. Log in to my bank.
4. Click Pay > choose ‘credit card’ from my ‘stored contacts’ > paste that amount.
5. Wait for 2FA code to arrive via SMS.
6. Paste 2FA code.
Yes please. I’d like to be able to script my life like this, such that the AI can adapt when my credit card provider changes their CSS class names or whatever would break my fragile Node script.
Can it also understand and solve captchas?
Many a time I’m surprised by how good Siri’s description of images is. I let Siri speak out some notifications when I’m wearing my EarPods:
- $person sent a picture of a group of people waiting at an airport.
- $person sent a screenshot of a social media website with an image and a caption.
- $person sent a picture of a dog sleeping on a wooden floor.
All of this to say: I’m not surprised they’re investing further into this.
- $person sent a picture of a group of people waiting at an airport.
- $person sent a screenshot of a social media website with an image and a caption.
- $person sent a picture of a dog sleeping on a wooden floor.
All of this to say: I’m not surprised they’re investing further into this.
For me the single best feature is pet detection in photos, 100% useless feature that nonetheless makes me happy (though I'm not sure how well it deals with lots of similar looking dogs - i.e. if you have a bunch of near identical dogs of the same breed+colour)