The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.
Cloud services, such as autoscaling EKS or AWS Batch are mostly limited by the GPU availability in a single region. That limits the scalability of jobs that can run distributedly in a large scale.
AI batch inference is one of the examples, and this post found that by going beyond a single region, it is possible to speed up the important embedding generation workload by 9x, because of the available GPUs in the "forgotten" regions.
This can significantly increase the iteration speed for building applications, such as RAG, and AI search. We share our experience for launching a large amount of batch inference jobs across the globe with the OSS project SkyPilot.
TL;DR: it speeds up the embedding generation on Amazon review dataset with 30M items by 9x and reduces the cost by 61%.
Dealing with all the Kubernetes pod configs / deployments is too much for an AI engineer. Being able to focus on the real model work would be super important.
The finetuning can tailor the model to have more customized knowledge, just like the identity knowledge of itself shown in the blog post. If you ask the original llama model, it should know nothing about SkyPilot or Vicuña, as it is trained on old knowledge from the internet.
However, finetuning still cannot get rid of the hallucination problem that all the chatbot suffers from.
It depends on how accurate you expect the chatbot should be. The retrieval might be considered more accurate, as it will not make up solutions, but just return irrelevant answer in the worst case.
Just want to add about hosting your own LLM vs using ChatGPT. Cost is definitely a thing to consider, but it also depends on whether it is ok to share the requests to your product with OpenAI.
Also, something you cannot do with ChatGPT is to custom it with your own data, such as internal documents, etc. As shown in the blog, the model trained by ourselves can easily know its identity.
SkyPilot is actually the tool that helps you find the resources on any cloud, including AWS, GCP, Azure, IBM (comming soon) or even Lambda Clouds. It can automatically search for the spot instances across all the regions and clouds, based on the availability and prices.
You need to use the transformers from the main branch instead of the pypi version, because the llama support is recently added. According to the readme of the repo, you need to install transformers with: pip3 install git+https://github.com/huggingface/transformers
If you follow this command in their instruction, the delta will be automatically downloaded and applied to the base model.
https://github.com/lm-sys/FastChat#vicuna-13b:
`python3 -m fastchat.model.apply_delta --base /path/to/llama-13b --target /output/path/to/vicuna-13b --delta lmsys/vicuna-13b-delta-v0`
It is mainly because of the legal issues caused by the license of llama model weights. We need to figure it out with Meta's llama team before releasing.