I'm actually working on a voice controlled, tldraw canvas based UI – and I'm a designer. So I feel quite seen by this article.
For my app, I'm trying to visualise and express the 'context' between the user and the AI assistant. The context can be quite complex! We've got quite a challenge to help humans keep up with reasoning and realtime models speed/accuracy.
Having a voice input and output (in the form of an optional text to speech) ups the throughput on understanding and updating the context. The canvas is useful for the user to apply spatial understanding, given that users can screen share with the assistant, you can even transfer understanding that way too.
I'm not reaching for the future, I'm solving a real pain point of a user now.
This is a tantalizing problem for me as a UX designer. My approach, which I'm quite proud of, places a UI primitive (Todo lists) center stage, with the chat thread on the side similar to Canvas or Claude's Artifacts. The interaction works like this:
1. User gets shown a list GUI based on their requirement (Meal Planning, Shopping List...)
2. Users speak directly to the list while the LLM listens in realtime
3. The LLM acknowledges with emojis that flash to confirm understanding
4. The LLM creates, updates or deletes the list items in turn (stored in localStorage or a Durable Object -> shout out https://tinybase.org/)
The lists are React components, designed to be malleable. They can be re-written in-app by the LLM, while still taking todos. The react code also provide great context for the LLM — a shared contract between user and AI. I'm excited to experiment with streaming real-time screenshots of user interactions with the lists for even deeper mind-melding.
I believe the cursor and chat thread remain critical. They ground the user and visually express the shared context between LLM and user. And of course, all these APIs are fundamentally structured around sequential message exchanges. So it will be an enduring UI pattern.
Do you have an idea for a voice-powered app? I can't stop having them.
- Imagine an app that reads recipes aloud while you're cooking
- Or one that helps manage your daily to-do list
- Or even one that gives you local weather reports
All while using your voice – and because it can talk, you don't even need to see it!
Building a voice-operated app is tricky – here's a video (https://www.youtube.com/watch?v=b61UaIpUbiE) of me at Atlassian explaining all the steps. You need to handle speech-to-text, voice activity detection, and text-to-speech. Everything needs to be fast and properly visualized for the user.
https://tinytalkingtodos.com/ is a workbench for creating voice-operated to-do lists that can talk back to you. It's a great way to mock up an app experience and test your idea. Here's how it works:
1. Select from over 25 templates in the app
2. Use the built-in code assistant to customize the template for your specific needs and desired look/feel
3. Write a system prompt for the assistant so it understands how your list should work
That's it! You can start capturing whatever you like. The app allows you to share your list with up to 5 people, and the contents sync instantly across all your devices. I think it's a great tool to help you experiment with multimodal AI and I hope you try it out.
I've been working on a voice assistant that runs in your browser, and persist data to localStorage using TinyBase. On mobile it's annoying when the screen switches off, you lose access to the assistant.
I wondered if using Audio Context, I could keep the session going in the background. I naively gave it a go, and with loads of assistance from Claude 3.5, it seems to actually work, and the whole main thread seems active? The UX seems novel to me, and I'm looking forward to experimenting more to understand the limits of background execution, but I'm surprised I can update my localStorage without looking at it. Would you expect this to work?
Meteor was/and is a fantastic community of very generous JavaScript developers. This is great news.
When it arrived and so important to me professionally. I used it to build my first web apps. I can't tell you how many times I struggled to get started with Rails, Meteor was a foothold beyond compare, they were incredibly focused on keeping the stack accessible to beginners.
They had free command line deployments – meant I could ship projects to friends, before I knew anything about hosting a web server, which was so exciting and really kept me motivated.
The way I do full-stack now, with declarative UI, apollo, Node.js – it's all pretty much the same idea as Meteor apps, just with even bigger communities that really out competed the meteor strack. You could say they're victims of JS explosion/success.
Now I have fantastic job as a full-stack dev, and I doubt I would of made it without Meteor and Discover Meteor PDF. So many cool projects like Apollo, Vue and Storybook came from people involved in the Meteor community.
There's still a huge gap in making accessible full-stack frameworks for web apps. This project is worth a look https://github.com/VulcanJS/Vulcan if you know I mean.
For my app, I'm trying to visualise and express the 'context' between the user and the AI assistant. The context can be quite complex! We've got quite a challenge to help humans keep up with reasoning and realtime models speed/accuracy.
Having a voice input and output (in the form of an optional text to speech) ups the throughput on understanding and updating the context. The canvas is useful for the user to apply spatial understanding, given that users can screen share with the assistant, you can even transfer understanding that way too.
I'm not reaching for the future, I'm solving a real pain point of a user now.
You can see a demo of it in action here -> https://x.com/ojschwa/status/1901581761827713134