If we can have this running locally on mobile phone that would be pretty cool. Imagine receiving a work document (for example, product requirement documents), and then this turning it into a podcast to play for me while I am driving. I think my productivity will be through the roof and I don't need to worry about compliance issues.
GGUF quantization is crucial for running models locally on devices, but quantizations can dramatically affect model's performance. It's essential to test models post-quantization (how benchmark comes in clutch). But we noticed a couple of challenges:
1. No easy, fast way to benchmark quantized GGUF models locally or on self-hosted servers.
2. GGUF quantization evaluation results in the existing benchmarks are inconsistent, showing lower scores than the official results from model developers.
Our Solution - We built a tool that:
1. Benchmarks GGUF models with one line of code.
2. Supports multiprocessing and 8 evaluation tasks.
3. In our testing, it's the fastest benchmark for GGUF models available.
Example:
Type below in terminal to benchmark Llama3.2-1B-Instruct Q4_K_M quant on the "ifeval" dataset for general language understanding. It took 80 minutes on a 4090 with 4 workers for multiprocessing.
We started with text models and plan to expand to more on-device models and modalities. Your feedback is welcome! If you find this useful, feel free to let us know on GitHub: https://github.com/NexaAI/nexa-sdk/tree/main/nexa/eval
Thanks for reporting. We are investigating this issue. Could you help submit an issue to our GitHub and provide a screenshot of the terminal (with pip show nexaai)? This could help us reproduce this issue faster. Much appreciated!
For anyone looking for a simple alternative for running local models beyond just text, Nexa AI has built an SDK that supports text, audio (STT, TTS), image generation (e.g., Stable Diffusion), and multimodal models! It also has a model hub to help you easily find local models optimized for your device.
It also comes with a built-in local UI to get started with local models easily and OpenAI-compatible API (with JSON schema for function calling and streaming) for starting local development easily.
You can run the Nexa SDK on any device with a Python environment—and GPU acceleration is supported!
Local LLMs, and especially multimodal local models are the future. It is the only way to make AI accessible (cost-efficient) and safe.
Currently, to modify the defined functions in the model, you can provide us with a description of your desired functions, and we can customize (fine-tune) a model for you.
However, if your use case is similar to the current functions of our model, you might consider using nested functions to leverage our existing capabilities.
Feel free to reach out for further assistance or more specific customization needs.
Our models offer developers faster, cheaper alternative to GPT-4o for implementing function-calling AI agent workflows. Developers can use our model APIs and implement each model's functions to integrate AI agent functionalities into their apps.
Octopus V4 proposes a graph of language models approach that creates a network of specialized open-source AI language models, where it acts as the master node coordinating with various domain-specific AI models as worker nodes. It aims to reduce the high computational, data, and energy demands that limit current large language model development and addresses restrictions imposed by proprietary models. In testing, such as the MMLU benchmark, Octopus V4 has achieved scores that suggest it can compete with, and potentially exceed, the performance of proprietary models, thus showing potential to match or surpass proprietary models through a collaborative, open-source approach.
The updated GGUF and safetensors will be released after final alignment tweaks. Please feel free to let us know if there's any other feedback!