The problem with C++ vs. unsafety is that there is really no boundary: All code is by default unsafe. You will need to go to great lengths to make it all somewhat safe, and then to even greater lengths to ensure any libraries you use won't undermine your safety.
In Rust, if you have unsafe code, the onus is on you to ensure its soundness at the module level. And yes, that's harder than writing the corresponding C++, but it makes the safe code using that abstraction a lot easier to reason about. And if you don't have unsafe code (which is possible for a lot of problems), you won't need to worry about UB at all. Imagine never needing to keep all the object lifetimes in your head because the compiler does it for you.
That very much depends on the code you're compiling. The factors that come into play are monomorphization (which means the compiler builds one copy of the code per type it is called for), procedural macros (which need to be fully compiled before being able to expand code using them), whether complex type shenanigans are used, etc. etc. Absent that, Rust will compile roughly as fast as C nowadays.
There is so much wrong with this article. Throw a little bit of ML pixie dust on everything for more hype? Check. Compare wildly different things as if they were the same? Double check.
digdugdirk has the right idea, and AFAIR, there is some work on that front (https://www.fornjot.app/).
Also the Fiat 500 goes 100km on about 4l of gas, while the Ford F150 uses 7l. No clue where the author gets the idea that the Fiat would get worse mileage, perhaps he's dividing by weight?
Very nice diagrams, they make the article really easy to follow! This really drives the point home that vector search isn't only a qantitative (as in faster), but a qualitative evolutionary step.
Makes one wonder what other use cases are lurking that would need just another small modification and haven't even been thought of yet because they used to be impossible to implement.
I totally agree that latency of this solution leaves a lot of room to improvement. But that's totally besides the point of the article, which is that people can get a no-cost semantic search for their personal website using those services. They can also use other solutions, of course.
Also I'm experimenting in further integrating things to reduce latency and most likely will publish another article within the month. Stay tuned.
Finally I somewhat agree that many of the players in the vector DB space try to push their cloud offerings. Which is fine, how else should they make money? And if latency matters that much to you, Qdrant offers custom deployments, too. I believe running Qdrant locally will handily beat your LanceDB solution perf-wise unless you're talking about less than 100k entries. We have both docker containers and release binaries for all major OSes, why not give it a try?
I also found while writing the article but after I had already done my research that cargo-lambda has grown some additional functionality that could have removed the need for the AWS CLI, but I wanted to get the article out, so I didn't test-drive that.
Thank you for the suggestion! I actually thought about using terraform, but I wanted to keep the experiment somewhat minimal regarding technologies and as I had already added AWS Lambda and Rust to the tech stack, I wanted to stay as close to the metal as possible. Besides, this is not for commercial applications, so I don't think high availability is in scope.
When starting this project I thought the same thing, but having done it I honestly cannot tell that much of a difference. Yes, there are two more steps in setting up the Lambda function, but in the end you still write an HTTP server and have them serve it.
Author here. This was a fun exercise to produce a semantic search using only free-tier services (which in the case of Cohere means non-commercial use only and in the case of AWS Lambda is limited to one year).
It also marks my first foray into using cloud services for a project. I've long been a cloud sceptic, and doing this confirmed some of my suspicions regarding complexity (mostly the administrative part around roles and URLs), while the coding part itself was a blast.
Nick's articles are always a delight to read, and this one is no exception. Openly discussing failure is one part, but coming up with so many ideas and having the gumption to actually try them out on a compiler is really fascinating.
Well, if the programmer writes the type annotation to actually have their code checked, how would a gradually typed language differ in practice from a static language?
Yes, you can still do dynamic typing. But I'd argue that using `dyn Any` you can do so in Rust, which is a statically typed language if I ever saw one.
Otherwise I completely agree about the languages growing closer together.
Yeah, that one hasn't aged too well. We've all seen the backswing to statically typed languages. Yes, some of them (e.g. typescript) run on top of dynamic langs, or allow for VMs (hi, WASM!). Why? Because ironically the same famed flexibility that makes it oh so easy to whip up a prototype is biting us in the ass when it comes to make a production-grade piece of software while staying on top of the ever changing requirements. So while we have a lot of python in ML (where most things haven't left prototyping stage), a lot of code nowadays is written in languages like Rust, TypeScript, Swift and others.
Not (only) because that's faster to run, but because it's faster to change while still remaining somewhat working correctly. And the current crop of compilers not only can produce stunningly fast code, but also awe-inspiringly great error messages that put the 90's and oughties' cryptic error messages to shame. Try that with a dynamic language!
What would that buy you? You train until you push your loss under a certain threshold, then check the external criteria and if they don't hold you train again? Your external criteria would essentially become another part of your loss function, but the whole training would become vastly more inefficient.
I'm saying that we shouldn't expect the models to come up with things we didn't train them to come up with.
Again, there is a lot of words to describe the fact that machine learning is just lossy compression for a bunch of data with the possibility to interpolate between data points and get somewhat plausible results. This means data points may get lost during compression/training, and certain things will look off, whether it be a preference for banana pairs, even numbers of fingers or certain weasel words in verbiage.
Having seen state of the art neural networks in the 70s and 80s, I am both in awe at the progress that has been made since and deeply unconcerned about AGI.
Yes, storage capacity and compute have made major leaps, but it's still only lossily compressing a semantic space. It won't generate things from thin air, and it's certainly not sentient, no matter how far you scale it up.
That said, it may be a useful tool in some cases, and we've also seen its numerous limitations, only some of which may be alleviated in the future.
As always, the problem is not artificial intelligence, but natural stupidity.
I sometimes wonder what CPU designers think when building such weird instructions? They must have some programs in mind that could be run faster with them, or else the additional transistors are just lost weight. But then compilers and language runtimes might or might not use those instructions. Add to that the fact that modern CPUs are basically their own compilers (going from "machine code" to microcode) and you have weirdness atop of more weirdness. But perhaps this is just for business sake; adding more instruction sets to provide a barrier to the competition, because programs using this run faster but are no longer portable to competitors' CPUs.
In Rust, if you have unsafe code, the onus is on you to ensure its soundness at the module level. And yes, that's harder than writing the corresponding C++, but it makes the safe code using that abstraction a lot easier to reason about. And if you don't have unsafe code (which is possible for a lot of problems), you won't need to worry about UB at all. Imagine never needing to keep all the object lifetimes in your head because the compiler does it for you.