maheshvaikri99

قبل 5 أشهرdiscuss

Hey everyone,

I've been working on SnapLLM for a while now and wanted to share it with the community. The problem: If you run local models, you know the pain. You load Llama 3, chat with it, then want to try Gemma or Qwen. That means unloading the current model, waiting 30-60 seconds for the new one to load, and repeating this cycle every single time. It breaks your flow and wastes a ton of time.

What SnapLLM does: It keeps multiple models hot in memory and switches between them in under 1 millisecond (benchmarked at ~0.02ms). Load your models once, then snap between them instantly. No more waiting. How it works: Built on top of llama.cpp and stable-diffusion.cpp Uses a vPID (Virtual Processing-In-Disk) architecture for instant context switching Three-tier memory management: GPU VRAM (hot), CPU RAM (warm), SSD (cold) KV cache persistence so you don't lose context

What it supports: Text LLMs: Llama, Qwen, Gemma, Mistral, DeepSeek, Phi, Unsloth AI models, and anything in GGUF format Vision models: Gemma 3 + mmproj, Qwen-VL + mmproj, LLaVA Image generation: Stable Diffusion 1.5, SDXL, SD3, FLUX via stable-diffusion.cpp OpenAI/Anthropic compatible API so you can plug it into your existing tools Desktop UI, CLI, and REST API

Model switch time between any of these: 0.02ms Getting started is simple: Clone the repo and build from source Download GGUF models from Hugging Face (e.g., gemma-3-4b Q5_K_M) Start the server locally Load models through the Desktop UI or API and point to your model folder Start chatting and switching

NVIDIA CUDA is fully supported for GPU acceleration. CPU-only mode works too.

With SLMs getting better every month, being able to quickly switch between specialized small models for different tasks is becoming more practical than running one large model for everything. Load a coding model, a medical model, and a general chat model side by side and switch based on what you need.

Ideal Use Cases: Multi-domain applications (medical + legal + general) Interactive chat with context switching Document QA with repeated queries On-Premise Edge deployment Edge devices like drones, self-driving vehicles, autonomous vehicles, etc Multi-agent workflow

Demo Videos: SnapLLM Desktop App Demo (Vimeo): https://vimeo.com/1157629276 SnapLLM Server and API Demo (Vimeo): https://vimeo.com/1157624031

The server demo walks through starting the server locally after cloning the repo, downloading models from Hugging Face, and loading them through the UI.

Links: GitHub: https://github.com/snapllm/snapllm Arxiv Paper: https://arxiv.org/submit/7238142/view

Star this repository - It helps others discover SnapLLM

MIT licensed. PRs and feedback welcome. If you have questions about the architecture or run into issues, drop them here or open a GitHub issue.

maheshvaikri99

قبل 6 أشهرdiscuss

Published the Benchmark with Results

https://ison.dev/benchmark.html

https://github.com/maheshvaikri-code/ison/tree/main/benchmar...

maheshvaikri99

قبل 7 أشهرdiscuss

Token Efficiency

  | Format       | Tokens | vs JSON  |
  |--------------|--------|----------|
  | ISONGraph    | 639    | -69%     |
  | ISON         | 685    | -66%     |
  | TOON         | 856    | -58%     |
  | JSON Compact | 1,072  | -47%     |
  | JSON         | 2,039  | baseline |

  LLM Accuracy

  | Format       | Correct | Accuracy | Acc/1K Tokens |
  |--------------|---------|----------|---------------|
  | ISONGraph    | 46/50   | 92.0%    | 143.97        |
  | ISON         | 44/50   | 88.0%    | 128.47        |
  | JSON         | 42/50   | 84.0%    | 41.20         |
  | JSON Compact | 41/50   | 82.0%    | 76.49         |
  | TOON         | 40/50   | 80.0%    | 93.46         |

  Key Findings

  1. ISONGraph wins on both efficiency AND accuracy - 92% correct with fewest tokens
  2. ISON/ISONGraph excel at multi-hop queries - LLM can follow relationships easily
  3. Acc/1K metric shows ISONGraph provides 3.5x more value per token than JSON
  4. Graph-specific format helps LLM understand relationships better than flat JSON

maheshvaikri99

قبل 7 أشهرdiscuss

Fair enough. Let me clarify the use case:

ISON isn't meant to replace JSON in your application. Your JS/TS code still uses JSON objects internally. ISON is specifically for the LLM context window.

The flow: App (JSON) → serialize to ISON → inject into prompt → LLM reasons → response → your app

You're right that nesting is lost. But for LLM reasoning, flat structures often work better. LLMs struggle with deeply nested JSON - they lose track of parent-child relationships 4+ levels deep.

On "tokens are getting cheaper": True for API costs. But context windows are still limited. When you're stuffing RAG results, memory, agent state, and user history into 128K tokens, every byte matters. It's not about saving money - it's about fitting more context.

On "wrong optimization": I ran the benchmark. Same data, same task. ISON: 88.3% accuracy. JSON: 84.7%. The LLM actually performed better with the tabular format, not just "equivalent for fewer tokens."

## BENCHMARK STATS:

TOKEN EFFICIENCY: ISON: 3,550 tokens JSON: 12,668 tokens

  ISON vs JSON:        72.0% reduction

LLM ACCURACY (300 Questions): ISON: 265/300 ( 88.3%) JSON: 254/300 ( 84.7%)

EFFICIENCY (Acc/1K): ISON: 24.88 JSON: 6.68 ISON is 272.3% MORE EFFICIENT than JSON!

But I hear you - if your data is deeply nested and that nesting carries semantic meaning the LLM needs, JSON might be the right choice. ISON works best for relational/tabular data going into context.

maheshvaikri99

قبل 7 أشهرdiscuss

Full raw benchmark logs published: https://github.com/AceIQ360/AceIQ360-Benchmark

Unlike others who only share top-line numbers, I published every question, answer, and judgment. You can verify the 100% claim yourself.

maheshvaikri99

قبل 7 أشهرdiscuss

Yes - I ran a 300 Questions benchmark comparing ISON vs JSON vs JSON-COMPACT etc on the same tasks.

ISON: 88.3% accuracy JSON: lower (can share exact numbers if interested)

Tested across Claude, GPT-4, DeepSeek, and Llama 3.

The key finding: LLMs handle tabular formats natively because they've seen billions of markdown tables and CSVs in training. No special prompting needed.

For associations, I tested with multi-table ISON docs like:

table.users id name 1 Alice 2 Bob

table.orders id user_id product 101 :1 Widget 102 :2 Gadget

Prompt: "What did Alice order?"

All models correctly resolved :1 → Alice → Widget without explicit instructions about the reference syntax.

The 30-70% token savings come from removing JSON's structural overhead (braces, quotes, colons, commas) while keeping the same semantic density.

Haven't published formal benchmarks on this yet - that's good feedback. I should.

maheshvaikri99

قبل 7 أشهرdiscuss

Ha, fair. CSV gets you 80% there.

The 20% ISON adds: - Multiple named tables in one doc - Cross-table references - No escaping hell (quoted strings handled cleanly) - Schema validation (ISONantic)

If you're stuffing one flat table into context, CSV works fine. When you have users + orders + products with relationships, ISON saves you from JSON's bracket tax.

maheshvaikri99

قبل 7 أشهرdiscuss

Essentially yes, but with a few additions CSV lacks:

1. Multiple tables in one document (table.users, table.orders) 2. References between tables (:user:42 links to id 42) 3. Object blocks for config/metadata 4. Streaming format (ISONL) for large datasets

The type annotations are optional - they help LLMs understand the schema without inference.

You could think of it as "CSV that knows about relationships" - which is exactly what multi-agent systems need when passing state around.

maheshvaikri99

قبل 7 أشهرdiscuss

Hi Thanks

Benchmark results and methodology here: https://github.com/AceIQ360/AceIQ360-Benchmark

The full system isn't open source yet - still deciding on licensing. But the benchmark repo has: - Complete results (500/500 on LongMemEval) - Raw logs showing each question/answer - Comparison with baselines

Happy to answer questions about the approach. The core insight: intelligent context organization beats raw context volume. No LLM calls for memory extraction - pure embedding-based retrieval using RudraDB (https://rudradb.com).

If you want to verify independently, I can provide API access.

maheshvaikri99

قبل 7 أشهرdiscuss

Fair point, but I'd push back on "none of these alternative formats exist in training data."

ISON isn't inventing new syntax. It's CSV/TSV with a header - which LLMs have seen billions of times. The table format:

table.users id name email 1 Alice [email protected]

...is structurally identical to markdown tables and CSVs that dominate training corpora.

On the "3x translation overhead" - ISON isn't meant for LLM-to-code interfaces where you need JSON for an API call. It's for context stuffing: RAG results, memory retrieval, multi-agent state passing.

If I'm injecting 50 user records into context for an LLM to reason over, I never convert back to JSON. The LLM reads ISON directly, reasons over it, and responds.

The benchmark: same data, same prompt, same task. ISON uses fewer tokens and gets equivalent accuracy. Happy to share the test cases if you want to verify.

maheshvaikri99

Submissions

SnapLLM: Switch between local LLM in under 1ms Multi-model&-modal serving engine

Show HN: ISON – Data format that uses 30-70% fewer tokens than JSON for LLMs

Show HN: MAPLE – Multi-agent protocol with resource-aware scheduling

Show HN: AceIQ360 – First AI memory system to achieve 100% on LongMemEval

comments