Bodhi App is an open-source local LLM inference solution that takes a different and simpler approach. Instead of re-inventing the wheel, it leverages existing, tried and tested ecosystem and solutions:
## Technical Architecture:
- llama.cpp as inference engine
- Rust/Axum backend for type-safe API layer
- Tauri for multiplatform builds
- HuggingFace integration
- YAML based configurations and update at runtime (no restarts required)
- OpenAI/Ollama API compatibility layer
## Key Technical Decisions:
1. No proprietary model format - directly use of GGUF files from HuggingFace
2. Opt-in Authentication, provides RBAC for team access
3. API design with proper authentication/authorization
4. Built-in Swagger UI with complete OpenAPI specs
5. Built-in User guide
# What Sets It Apart:
Designed with non-technical users in mind. So it comes a basic Web-based user interface, allowing users to get started quickly with their first AI-assistant conversation.
## Setup Wizard:
- App displays a setup wizard when run for first time
- Allows user to download popular models in a user friendly way
## Built-in Chat UI:
- Ships with a complete Chat UI
- Chat UI is simple enough for non-technical users to get started with their first AI-conversation
- Adapts to power users by providing complete control over request settings
Bodhi App is an open-source local LLM inference solution that takes a different and simpler approach. Instead of re-inventing the wheel, it leverages existing, tried and tested ecosystem and solutions:
## Technical Architecture:
- llama.cpp as inference engine
- Rust/Axum backend for type-safe API layer
- Tauri for multiplatform builds
- HuggingFace integration
- YAML based configurations and update at runtime (no restarts required)
- OpenAI/Ollama API compatibility layer
## Key Technical Decisions:
1. No proprietary model format - directly use of GGUF files from HuggingFace
2. Opt-in Authentication, provides RBAC for team access
3. API design with proper authentication/authorization
4. Built-in Swagger UI with complete OpenAPI specs
5. Built-in User guide
# What Sets It Apart:
Designed with non-technical users in mind. So it comes a basic Web-based user interface, allowing users to get started quickly with their first AI-assistant conversation.
## Setup Wizard:
- App displays a setup wizard when run for first time
- Allows user to download popular models in a user friendly way
## Built-in Chat UI:
- Ships with a complete Chat UI
- Chat UI is simple enough for non-technical users to get started with their first AI-conversation
- Adapts to power users by providing complete control over request settings
- Supports realtime streaming response, markdown rendering, code rendering with syntax highlights
- Displays chat stats, request tokens, response tokens, token speed
- Allow copying of the AI-response etc.
## Built-in UI for Model + App Management + API access:
- Manage complete Model lifecycle from the UI
- Downloading models, deleting models
- Configuring models, request + inference server configurations using Model Alias yaml files
- Allows configuring for parallel processing of requests
- Configuring App Settings - chosing betwen CPU/GPU, server idle time etc.
- API tokens for authenticated/authorized access to APIs by 3rd party
## Tech for UI:
- Uses Nextjs, Tailwindcss, Shadcn to build powerful, responsive and user friendly UI
- Supports Dark/Light mode
- Exported using config `output: "export"` to export the entire frontend as static html + javascript
- Served by the backend as static asset
- Thus no packaged nodejs server, reducing app size, complexity and compute
# Links
Try it out: https://getbodhi.app/
Source: https://github.com/BodhiSearch/BodhiApp
Looking forward to technical feedback and discussions.