It is inference latency most of the time. These VLA models take in an image + state + text and spit out a set of joint angle deltas.
Depending on the model being used, we may get just one set of joint angle deltas or a series of them. In order to be able to complete a task, it will need to capture images from the cameras, current joint angles and send them to the model along with the task text to get the joint angle changes we will need to apply. Once the joint angles are updated, we will need to check if the task is complete (this can come from the model too). We run this loop till the task is complete.
Combine this with the motion planning that has to happen to make sure the joint angles we are getting do not result in colliding with the surroundings and are safe, results in overall slowness.
I have been working with LLMs and VLMs to automate browser based workflows among other things for the last couple of years. Given how good the vision models have gotten lately, the perception problem is solved to level where it opens up a lot of possibilities. Manipulation is not generally solved yet but there is a lot of activity in the field and there are promising approaches to solve (OpenVLA, π0). Given these, I'm trying to build an affordable robot that can help around with household chores using language and vision models. Idea is to ship capable enough hardware that can do a few things really well with the currently available models and keep upgrading the AI stack as manipulation models get better over time.
For now it still uses openai for embeddings generation by default and we are updating that in the next couple of releases to be able to use a local model for embedding generation before writing to a vector db.
Disclosure: I'm the maintainer of LLMStack project
If anyone is looking to try it out quick without local installation, we added Llama-8B model to Promptly playground. Please check it out at https://trypromptly.com/playground.
We can get a lot done with vector db + RAG before having to finetune or custom models. There are a lot of techniques to improve RAG performance. Captured a few of them a while back at https://llmstack.ai/blog/retrieval-augmented-generation.
We have recently added support to query data from SingleStore to our agent framework, LLMStack (https://github.com/trypromptly/LLMStack). Out of the box performance performance when prompting with just the table schemas is pretty good with GPT-4.
The more domain specific knowledge needed for queries, the harder it has gotten in general. We've had good success `teaching` the model different concepts in relation to the dataset and giving it example questions and queries greatly improved performance.
Another project I worked on for my own use was a network isolated, lightweight video monitoring system. Around 5 years ago, I was looking to install a camera in our living room. I couldn't find anything I trusted that worked completely offline without some companion app pinging their servers. So I bought a basic IP camera on Amazon that supports rtsp and a raspberry pi. Created a fenced wifi network and added the camera to it.
Had an FFmpeg process read camera stream on demand and write to local buffers. Wrote a simple python server to listen for incoming connections on a different interface and stream the video on API requests. Then built an android app that talks to the python server to stream video on demand.
Also installed motion (https://github.com/Motion-Project/motion) on raspberry pi to detect motion in the video and store those snippets to local storage. With motion running, the adapter I was using wasn't delivering enough power resulting in storage occasionally unmounting and raspberry pi restarting taking the camera system offline. With motion detection disabled, the entire setup ran reliably for many years.
This will hopefully improve the startup times for FFmpeg when streaming from virtual display buffers. We use FFmpeg in LLMStack (low-code framework to build and run LLM agents) to stream browser video. We use playwright to automate browser interactions and provide that as tool to the LLM. When this tool is invoked, we stream the video of these browser interactions with FFmpeg by streaming the virtual display buffer the browser is using.
There is a noticeable delay booting up this pipeline for each tool invoke right now. We are working on putting in some optimizations but improvements in FFmpeg will definitely help. https://github.com/trypromptly/LLMStack is the project repo for the curious.
I've been using Django as my main choice for web projects for over ten years. The reason I like it so much is because it comes with a lot of built-in features that one needs to ship web projects to production. For example, I was first attracted to Django because of its admin interface and its straightforward views and templating system.
Over the years, Django has kept up with changes in web development. An example of this is when database migrations, which used to be a separate project, were integrated into Django itself. The Django community is also strong with great ecosystem projects like DRF for APIs, Django Channels for real-time features, and social-auth for social sign-ins.
My recent use of Django is in (https://github.com/trypromptly/LLMStack). We use Django Channels for WebSocket support, DRF for APIs, and ReactJS for the frontend.
We built https://github.com/trypromptly/LLMStack to serve exactly this persona. A low-code platform to quickly build RAG pipelines and other LLM applications.
Kudos to the team for a very detailed notebook going into things like pipeline evaluation wrt performance and costs etc. Even if we ignore the framework specific bits, it is a great guide to follow when building RAG systems in production.
We have been building RAG systems in production for a few months and have been tinkering with different strategies to get the most performance out of these pipelines. As others have pointed out, vector database may not be the right strategy for every problem. Similarly there are things like lost in the middle problems (https://arxiv.org/abs/2307.03172) that one may have to deal with. We put together our learnings building and optimizing these pipelines in a post at https://llmstack.ai/blog/retrieval-augmented-generation.
https://github.com/trypromptly/LLMStack is a low-code platform we open-sourced recently that ships these RAG pipelines out of the box with some app templates if anyone wants to try them out.
There are a lot of things that goes on in production to scale any service to handle that level of requests. But Django as a web framework is good at what it does. It comes in with most things that one needs to put web apps in production.
We recently open-sourced an LLM apps platform (https://github.com/trypromptly/LLMStack) that is entirely built with django as backend (drf for APIs, channels for websockets and reactjs for frontend).