I just made a new installer for Discourse on CharmRuby, now I gotta check this out and see if porting is feasible. Hopefully this reduces the app size, that is quite large with CharmRuby
First time I was in San Francisco and someone introduced themselves like that, going even beyond, was indeed a super weird experience being a brazilian.
It's the Amazon own model. I'm baffled someone would pick it, even more that someone would test Llama 4 for a task in an age where Sonnet 4.5 is already out, so in the last 45 days.
Looks like they were limited by AWS Bedrock options.
> what does the rag for uploaded files do in discourse?
You can upload files that will act as RAG files for an AI bot. The bot can also have access to forum content, plus the ability to run tools in our sandboxed JS environment, making it possible for Discourse to host AI bots.
> also, when i run a discourse search does it really do both a regular keyword search and a vector search? how do you combine results?
Yes, it does both. In the full page search it does keyword first, then vector asynchronously, which can be toggled by the user in the UI. It's auto toggled when keyword has zero results now. Results are combined using reciprocal rank fusion.
In the quick header search we simply append vector search to keyword search results when keyword returns less than 4 results.
> does all discourse instances have those features? for example, internals.rust-lang.org, do they use pgvector?
We host thousands of forums but each one has its own database, which means we get a sort of free sharding of the data where each instance has less than a million topics on average.
I can totally see that at a trillion scale for a single shard you want a specialized dedicated service, but that is also true for most things in tech when you get to the extreme scale .
I was taken back when I saw what was basically zero recall loss in the real world task of finding related topics, by doing the same thing you described where we over capture with binary embeddings, and only use the full (or half) precision on the subset.
Making the storage cost of the index 32 times smaller is the difference of being able to offer this at scale without worrying too much about the overhead.
Depends on your needs. You surely don't want 32k long chunks for doing the standard RAG pipeline, that's for sure.
My use case is basically a recommendation engine, where retrieve a list of similar forum topics based on the current read one. As with dynamic user generated content, it can vary from 10 to 100k tokens. Ideally I would generate embeddings from an LLM generated summary, but that would increase inference costs considerably at the scale I'm applying it.
Having a larger possible context out of the box just made a simple swap of embeddeding models increase quality of recommendations greatly.
Just migrated all embeddings to this same model a few weeks ago in my company, and it's a game changer. Having 32k context is a 64x increase when compared with our previous used model. Plus being natively multilingual and producing very standard 1024 long arrays made it a seamless transition even with millions of embeddings across thousands of databases.