I modified the OpenAI Realtime API Console Demo App to add a map to it, and I connected the map events to the Realtime API.
Result... Im able to move around the map and talk to it! Its fascinating how its able to reverse geocode out of memory, Ideally I should add tools to reverse Geocode the locations. Im planning to add more tools to it like internet speed locations and maybe weather
I built a voice-controlled map where an AI can see exactly where you're looking and describe those locations with the OpenAI Realtime API.
Its a very simple setup where, as you pan and zoom around the map, the AI receives your current coordinates, zoom level, and visible area. Now I can add tools like get the internet speed or weather for that area and I will have a Voice Controlled Map.
I was using transformer.js to generate and use embeddings with small models in the browser itself, its quite useful to implement any kind of semantic search.
Result... Im able to move around the map and talk to it! Its fascinating how its able to reverse geocode out of memory, Ideally I should add tools to reverse Geocode the locations. Im planning to add more tools to it like internet speed locations and maybe weather
https://github.com/milind-soni/realtime-map-chat
Check out here if you would like to use the code