@irfn - that's an interesting idea. will definitely try to create benchmark using my local M2 machine and llama3-7b, just for comparison.
yes, ollama and Bodhi App both use llama.cpp but our approaches are different. Ollama embeds a binary within its binary, that it copies to a tmp folder and runs this webserver. any request that comes to ollama is then forwarded to this server, and replied sent back to the client.
Bodhi embeds the llama.cpp server, so there is no tmp binary that is copied. when a request comes to Bodhi App, it invokes the code in llama.cpp and sends the response back to client. So there is no request hopping.
Hope that approach do provide us with some benefits.
Also Bodhi uses Rust as programming language. IMHO rust have excellent interface with C/C++ libraries, so the C-code is invoked using the C-FFI bridge. And given Rust's memory safety, fearless concurrency and zero cost abstractions, should definitely provide some performance benefit to Bodhi's approach.
Will get back to you once I have results for these benchmarks. Thanks for the idea.
Hope you try Bodhi, and have some equally valuable feedback on the app.
you cannot run gumroad like financial transaction business without licence and agreement with stripe. it is bound to be closed down on short notice, and all the earnings will be clawed back.
One prefers locking and complicated instructions, the other is designed with open and simplicity.
At one point, Intel seemed to be dominant, and proven right with their proprietary tech, where as ARM chugged along keeping a low profile, and keep doing the good work. Now, finally the reality has caught up with them, and turns out openness and simplicity is finally taking a lead.
With LLMs/ChatGPT, i would prefer fine-tuning a model and have it auto-generate bash commands with my natural language description.
I remember, I used to keep a diary with landline phone numbers of relatives etc. I used to remember at least a dozen of those numbers for emergency. With mobile and digital contacts, now that number is down to just 2.
Wonder, what the programming is going to look like in 10 yrs. If no-one dedicates so much of their time and effort to document, the programmers then would feel helpless. right?
1. Or at that point, LLMs will be intelligent enough to read the source code and figure out the commands.
2. They can generate data by running commands, and train themselves using this simulated data.
What other scenarios will occur once we stop generating data using human effort because we all are so dependent on LLMs to generate those commands?
we need ML GPU workload standardization, and more options to chose from. Sucks being held hostage by Nvidia. No ML GPUs available in my country, and waitlist goes in years.
read sapiens and other books by yuval noah harari, and later there were articles that many things he claimed in his book were incorrect, or later proven to be incorrect. always blows my mind that we still just have theories of how humans spread across the world, and how the languages and culture developed. if the human development and migrations are clarified definitely, many of the regional conflicts will end.
Each company have different motivations. For e.g. Basecamp moved to save cost.
Working in Enterprise, I can share hybrid model works best for multinational. If they are going into new geography, where local laws restrict data transfer across national boundaries, having a cloud provider with local data centers is the easiest and quickest way to launch.