The article’s distinction about memory is a bit too simplified. Virtual memory is address space mapped or reserved by the process. RSS is resident physical pages, not necessarily “actively used.”. RSS also overcounts shared memory, which is why PSS/USS matter.
It's quite exciting how far we've come from the modern exposition of world models by David Ha and Jürgen Schmidhuber in 2018 https://worldmodels.github.io/
I don't think this is bad, if you know Python then most of the code will be fine for you. I think you're probably referring to pm_reduce_collapse, but while it looks daunting at first, it really isn't when you consider the alternatives, I'd be curious on how you'd improve it short of creating a DSL.
I have written parsers using parser combinators in Haskell and Clojure. I find that ML-like (Haskell, OCaml, StandardML) languages generally are great at writing parsers, even hand-written ones in it is a superior experience.
In this case, this was a project at $EMPLOYER in an existing codebase with colleagues who have never seen Haskell code, using Haskell would've been a major error in judgement.
One anecdote in the same vein, a couple of months ago, I wanted to parse systemd-networkd INI files in Python and the python built-in ConfigParser [0] and pytest's iniconfig parser [1] couldn't handle multiple sections with the same name so I ended up writing 2 parsers, one using a ParserCombinator library and one by hand and ended up using the latter given it was much simpler to understand and I didn't have to introduce an extra dependency.
Admittedly, INI is quite a simple format, hence I mention this as an anecdote.
I have used systemd services before to do this to run an application, I had a user created specifically for the application, and I defined the capabilities the application needed via CapabilityBoundingSet and AmbientCapabilities [0] and I used a lot of stuff from [1] to restrict the application e.g. the sandboxing facilities, restricting the allowed syscalls [2], ...etc. systemd also comes with a useful command systemd analyze security [3]
10 years ago I've implemented SCAMP (a gossip protocol) in Clojure, you might find it interesting, the implementation is quite small https://github.com/cipherself/gossip
Assuming that's what he meant, why would it be considered baseline versus anything else? I am genuinely curious because I'd like to know more about issues people face with this or that vector store in general.
Got it, I think this might make sense for a "conversation" type of search not for an instant search feature because lowest latency is gonna be too high IMO.
I am working on search but rather for text-to-image retrieval, nevertheless, I am curious if by that's all baked into Azure AI search you also meant synthetic query generation from the grandparent comment. If so, what's your latency for this? And do you extract structured data from the query? If so, do you use LLMs for that?
Moreover I am curious why you guys use bm25 over SPLADE?