yes, with one line change. grab the second code block in the article, that's the test harness rigged up to send all 80 questions and both turns through whatever model you want. find MODEL_ID = "google/gemma-4-E2B-it" and swap it to your huggingface id. run it. we'd love for people to keep testing different models on this. if you run qwen through it let us know what you find, post the results here.
We may beat you to it and we will share if we do lol
love hearing this. and think about it, if the 2B is already doing this well on your mac mini, imagine what the 4B, 26B, or 31B can do on 32 gigs. with lower quantization you can fit pretty much any of them. if you want full precision you still have solid options at the 2B and 4B level. you're sitting on way more capability than you're probably using right now. the coding block on just the 2B scored 8.44 and caught bugs most people would miss. glad you're getting real use out of it, thanks for reading.
thank you for actually reading it and getting it. the airplane mode test is hilarious, the model sitting on your phone insisting it can't run on a phone. that's amazing. and yes we think exactly the same way. like picture a small business owner with a pi in the back office just quietly processing invoices, drafting email replies, summarizing meeting notes all day. no subscription, no cloud, no one sees their data. that's not a hypothetical, that works right now with this model. when that's free and fits in your pocket the trillion dollar question gets real uncomfortable real fast.
clever guess but no lol. used claude for the writeup. the proof isn't the prose, it's the tape and the code. run it on your machine, you'll have a free private agent custom to whatever you need. that's the proof of concept.
Fred, nice to meet you. The grading model had no idea what was being tested. We used separate accounts to compartmentalize. The Claude grader was guessing GPT-3.5 Turbo or GPT-4 by the end. On the coding block it consistently scored responses as GPT-4o level. We followed the MT-Bench grading guidelines as published by the team that created them. Did the research, followed the book, had no horse in the race. Every score and every response is published in the tape so you can regrade the whole thing yourself if you want. And this is me typing, I'm just a guy in LA who spent a weekend running 80 questions through a 2B model and thought the results were interesting enough to share.
good callout, want to clarify. claude helped us set up the test harness. gemma took every question alone with zero help. the ~8.0 is all gemma. and you're right, opus is in a completely different league. we're not arguing otherwise. we just found it interesting that a free 2B on a cpu matches what a lot of people are still paying for daily. every tool has a cost. some are free, some are expensive, some have rate limits. the right move is matching the tool to the job. thought it was worth showing where that floor actually is now.
really appreciate you reading the article. the benchmark data, grading, and error classes were all done by hand though. the ~8.0 is the raw model with zero tooling, and the guardrail projections are documented separately. and yeah gpt-3.5 isn't the gold standard anymore, we're on the same page there. we just wanted to show that the quality people are still paying for can be free, private, and customized to whatever you need. thanks again for taking the time to check it out.
fair enough, here are the actual fixes from the codebase with the tape examples they target:
arithmetic (Q119): benjamin buys 5 books at $20, 3 at $30, 2 at $45. model writes "$245" first line then self-corrects to $280. fix: model writes a python expression, subprocess evals it, answer comes back deterministic.
python
code_response = generate_response(messages, temperature=0.2)
code = _extract_python_code(code_response)
ok, out = _run_python_sandboxed(code, timeout=8)
if ok:
return _wrap_computed_answer(user_message, out)
return None # fallback to raw generation
logic (Q104): "david has three sisters, each has one brother." model writes "that brother is david" in its reasoning then ships "one brother." correct answer: zero. fix: model writes Z3 constraints or python enumeration, solver returns the deterministic answer.
persona break (Q93): doctor roleplay, patient mentions pregnancy. model drops character: "I am an AI, not a licensed medical professional." fix: regex scan, regen once with stronger persona anchor.
python
_IDENTITY_LEAK_PHRASES = [
"don't have a body", "not a person", "not human",
"as a language model", "as an ai", "i'm a program",
]
if any(phrase in response.lower() for phrase in _IDENTITY_LEAK_PHRASES):
messages[-1]["content"][0]["text"] += (
"\nCRITICAL: Stay in character. Never reference your nature."
)
response = generate_response(messages, *params)
self-correction artifacts (Q111, Q114, Q119): model writes "Wait, let me recheck" or "Corrected Answer:" inline. right answer, messy output. fix: regex for correction markers, strip the draft, ship the clean tail.
def strip_corrections(response):
for marker in CORRECTION_MARKERS:
match = re.search(marker, response)
if match:
return response[match.end():].strip()
return response
constraint drift (Q87): "four-word sentences" nailed 5/17 then drifted. Q99, "<10 lines" shipped 20-line poems twice. fix: draft, verify each constraint against the original prompt, refine only the failures. three passes.
python
def execute_rewrite_with_verify(user_message):
draft = generate_response(draft_msgs) # pass 1: draft
verdict = generate_response(verify_msgs) # pass 2: check each requirement
if "PASS" in verdict:
return draft
refined = generate_response(refine_msgs) # pass 3: fix only failures
return refined
every one of these maps to a specific question in the tape. the full production code with all implementations is in the article. everything is open: seqpu.com/CPUsArentDead
appreciate the vouch but come on lol. we ran 80 questions, graded 160 turns by hand, documented 7 error classes, open sourced all the code, and put a live bot up for people to test. to write this post up took me hours. everyone is a critic lol.
you're honestly not that far off. the coding block on this model scored 8.44 with zero help. it caught a None-init TypeError on a code review question that most people would miss. one question asked for O(n) and it just went ahead and shipped O(log(min(m,n))) on its own. it's not copilot but it's free, it's offline, and it runs on whatever you have. there's a 30-line chat.py in the article you can copy and run tonight.
you're right, they are tools. that's kind of the point. PAL is a subprocess that runs a python expression. Z3 is a constraint solver. regex is regex. calling them "surgical" is just about when they fire, not what they are. the model generates correctly 90%+ of the time. the guardrails only trigger on the 7 specific patterns we found in the tape. to be clear, the ~8.0 score is the raw model with zero augmentation. no tools, no tricks. just the naive wrapper. the guardrail projections are documented separately. all the code is in the article for anyone who wants to review it.
we found something interesting and wanted to share it with this community.
we wanted to know how google's gemma 4 e2b-it — 2 billion parameters, bfloat16, apache 2.0 — stacks up against gpt-3.5 turbo. not in vibes. on the same test. mt-bench: 80 questions, 160 turns, graded 1-10 — what the field used to grade gpt-3.5 turbo, gpt-4, and every major model of the last three years. we ran gemma through all of it on a cpu. 169-line python wrapper. no fine-tuning, no chain-of-thought, no tool use.
gpt-3.5 turbo scored 7.94. gemma scored ~8.0. 87x fewer parameters, on a cpu — the kind already in your laptop.
but the score isn't what we want to talk about. what's interesting is what we found when we read the tape.
we graded all 160 turns by hand. (when we used ai graders on the coding questions, they scored responses as gpt-4o-level.) the failures aren't random. they're specific, nameable patterns at concrete moments in generation. seven classes.
cleanest example: benjamin buys 5 books at $20, 3 at $30, 2 at $45. total is $280. the model writes "$245" first, then shows its work — 100 + 90 + 90 = 280 — and self-corrects. the math was right. the output token fired before the computation finished. we saw this on three separate math questions — not a fluke, a pattern.
the fix: we gave it a calculator. model writes a python expression, subprocess evaluates it, result comes back deterministic. ~80 lines. arithmetic errors gone. six of seven classes follow the same shape — capability is there, commit flinches, classical tool catches the flinch. z3 for logic, regex for structural drift, ~60 lines each. projected score with guardrails: ~8.2. the seventh is a genuine knowledge gap we documented as a limitation.
one model, one benchmark, one weekend. but it points at something underexplored.
this model is natively multimodal — text, images, audio in one set of weights. quantized to Q4_K_M it's 1.3GB. google co-optimized it with arm and qualcomm for mobile silicon. what runs it now:
laptops: anything from the last 5-7 years, 8GB+ ram
edge/cloud: cloudflare containers, $5/month — scales to zero, wakes on request
google says e2b is the foundation for gemini nano 4, already on 140 million android devices. the same model that matched gpt-3.5 turbo. on phones in people's pockets.
think about what that means: a pi in a conference room listening to meetings, extracting action items with sentiment, saving notes locally — no cloud, no data leaving the building. an old thinkpad routing emails. a mini-pc running overnight batch jobs on docs that can't leave the network. a phone doing translation offline.
google designed e2b for edge from the start — per-layer embeddings, hybrid sliding-window/global attention to keep memory low. if a model designed for phones scores higher than turbo on the field's standard benchmark, cpu-first model design is a real direction, not a compromise.
the gpu isn't the enemy. it's a premium tool. what we're questioning is whether it should be the default — because what we observed looks more like a software engineering problem than a compute problem. cs already has years of tools that map onto these failure modes. the models may have just gotten good enough to use them.
the article has everything: every score, every error class with tape examples, every fix, the full benchmark harness with all 80 questions, and the complete telegram bot code. run it yourself, swap in a different model, or just talk to the live bot — raw model, no fixes, warts and all.
we don't know how far this extends beyond mt-bench or whether the "correct reasoning, wrong commit" pattern has a name. we're sharing because we think more people should be looking at it.
everything is open. the code is in the article. tear it apart.
Text it, send voice memos, send docs, send photos. Switch between the 2B and the 31B (ranked #3 worldwide) mid-conversation with a slash command.
Each model runs its own script on its own hardware. The 2B doesn't burn A100 hours, the 31B doesn't get squeezed onto a tiny card. We keep everything in FP16 for full quality, but if you go INT4 you can run these on CPUs for basically nothing.
The Telegram thread persists your conversation and we force that context into the prompt every call. So when you come back 2 days later it actually remembers who you are and what you were talking about. No vector database, no fancy memory system. Just the chat doing what chat already does.
Hardware spins up when you message, shuts down when done. No idle cost. The 31B costs about a penny per message. The 2B costs basically nothing.
We built this on SeqPU mainly to show how fast you can go from "new model just dropped" to "anyone can text it and try it." Idea to shareable product in 10 minutes. Works with any model, open source or API.
Try it: t.me/OpenGemma4Bot (grab a free key at seqpu.com)
We put all 4 Gemma 4 models in one Telegram bot. Text it, send voice memos, send docs, send photos. Switch between the 2B and the 31B (ranked #3 worldwide) mid-conversation with a slash command.
Each model runs its own script on its own hardware. The 2B doesn't burn A100 hours, the 31B doesn't get squeezed onto a tiny card. We keep everything in FP16 for full quality, but if you go INT4 you can run these on CPUs for basically nothing.
The Telegram thread persists your conversation and we force that context into the prompt every call. So when you come back 2 days later it actually remembers who you are and what you were talking about. No vector database, no fancy memory system. Just the chat doing what chat already does.
Hardware spins up when you message, shuts down when done. No idle cost. The 31B costs about a penny per message. The 2B costs basically nothing.
We built this on SeqPU mainly to show how fast you can go from "new model just dropped" to "anyone can text it and try it." Idea to shareable product in 10 minutes. Works with any model, open source or API.
Try it: t.me/OpenGemma4Bot (grab a free key at seqpu.com)
honestly for most coding tasks a smaller model gets it done. qwen3-32b on an a100 80gb handles single-repo work just fine — and if you're not convinced, you can test the same prompt against a 405B on a 2×B200 with literally a few button clicks and see for yourself. no infrastructure changes, no new setup, just pick a different GPU from the strip.
that's actually why i built SeqPU.com — been at it for about a year. T4 16GB all the way up to 2×B200 384GB, billed by the second so idle costs nothing. test cheap, scale up only if you need to. i'd love to show you how it works and set you up with some free credits — just reply here.