Bradley-Terry and Elo scores are equivalent mathematical models! The fundamental presumption is the same Thurstone model - that an individual's skill in a particular game is a normally distributed random variable around their fundamental skill.
We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw), but we found that even better was to calculate Elo scores, do cross-query bias adjustment, and then MSE loss to predict the Elo score itself.
Yeah absolutely. In your link, it iterates on _ = ^{_}, until it finds the fixed point.
In our training pipeline, we had to convert the fixed point iteration to be on _ directly for numerical stability. I have a post on that here!: https://hackmd.io/x3_EkXGKRdeq-rNHo_RpZA
Bradley-Terry also very cleanly turns into a loss function that you can do gradient descent on, which will cause your model to efficiently learn Elo scores!
Our calculations are at: https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw
tldr; wikipedia iterates on <e^elo>, but that can go to zero or infinity. Iterating on <elo> stays between -4 and 4 in all of our observed pairwise matrices, so it's very well-bounded.
We found that MSE after elo-adjustment worked equally well. And, MSE lets you shuffle (q, d) across the dataset which has good statistical properties (Versus contrastive, which makes you sample the same query many times within a single minibatch)
In this case "InfoNCE" isn't applicable because the reranker's output is a scalar, not a vector. So that's why we checked both bradley-terry and MSE.
Yeah that's exactly what we observed. Our goal was to create an absolute score that's completely independent from the Corpus, which is difficult because naturally all ELO distributions are inherently tied to the corpus itself!
When we were exploring the mathematical foundations, we considered ELO scoring against a "Universal Corpus" based on the natural entropy of human language (Obviously that's intractable, but sometimes this term cancels out like in the DPO proof).
But eventually we figured out a method using cross-query comparisons to assign an "ELO bias" to all document ELOs within a given query's candidate list. This normalizes it correctly such that when a candidate list is all bad, the ELOs shift low. And when the candidate list is all good, the ELOs shift high. Even when the relative ELOs are all the same.
I often see it rendered as "Elo" but I've always found it more natural to capitalize as "ELO", but perhaps I should swap to "Elo" given this. Pronouncing "ee-low" is certainly the way it's done in chess/esports though!
Hey! We actually did a lot of research into ELO consistency, i.e. to check whether or not the NxN pairwise matrix followed the ELO model. It was a long road that's probably grounds for an entirely separate blog post, but the TLDR is that we observe that:
For each document, there is a secret hidden score "s" which is the "fundamental relevance according to the LLM". Then, when we sample (q, d1, d2) from the LLM, the LLM follows the statistical property that:
- The "fundamental hidden preference" is `pref = s_{d1} - s_{d2}`, usually ranging between -4 and 4.
- The LLM will sample a normal distribution around the `pref` with stddev ~0.2, which is some "inner noise" that the LLM experiences before coming to a judgement.
- The preference will pass through the sigmoid to get a sampled_score \in [0, 1].
- There is an additional 2% noise. i.e., 0.98 * sampled_score + 0.02 * random.random()
When we use Maximum Likelihood Estimation to find the most likely predicted "hidden scores" \hat{s} associated with each document, then we go ahead and sample pairwise matrices according to `0.98 * sigmoid( \hat{s}_1 - \hat{s}_2 + N(0, 0.02) ) + Uniform(0.02)`, then we get a pairwise matrix with virtually identical statistical properties to the observed pairwise matrices.
Yes, a step where you do a structured extraction into a database column would be a potential solution. But, it requires a preprocessing step.
It all depends on the use-case, sometimes you get a query that you couldn't have predicted the filter beforehand. In those cases, usually what you have to do is open up a spreadsheet and then manually categorize every document by hand. LLMs and modern AI are great ways to automate this.
A really good solution might be to have a system that computes these filters on-the-fly, but also caches them for later reuse if a query asks for that filter again.
For long documents we have a rolling window strategy. So, we cut the document into 5,000 token groupings for use in inference. There's also a 400 token overlap, and we prefer the earlier chunk for overlap tokens.
For example, if Group #0 overlaps with Group #1 at index 5,200, then we use the logprob from Group #0, because it had more context. Group #1 gets the benefit of context for indices 5,000-5,400, even though we toss out the logprobs for that range.
No need to keep track of chunks that it's already made, we just want heatmap values and then we use those heatmaps to split at the hottest character that's around our target chunk length (Or use a threshold value and binary search the threshold for our target # Chunks or average chunk size).
It scored better than LlamaIndex's recursive character text splitter and that was including some custom regex work to improve it. If you put enough effort into the regex you could probably get there, but the whole point of the agentic chunking is for it to be automatic and contextual.
I've written a lot of RAG pipelines over the last year, and one consistent pain-point is writing regex to chunk the documents correctly.
Right now, the most common chunking algorithms are:
- Split every 1000 characters
- Split on whitespace
- Recursively split on: (many newlines, then one newline, then periods, then spaces)
The best is recursive character text splitter, but regex is super brittle and when it fails to match it ends up creating huge chunks. Worse, this solution also has the overhead of needing to maintain regexes for every single filetype.
Here we propose LlamaChunk, an inference-efficient method of LLM-powered chunking. Using this method, it only requires a single LLM inference over your document in order to provide the most optimal recursive character text splitting, without needing to hope that a bunch of hard-coded rules work on your unstructured data.
That's because the $50M were owned by a large percentage of the users of Ethereum, not a single person. It represented 15% of the total ETH in circulation, back when the currency was in a very nascent state and the flagship product that ETH provided that BTC didn't, was the DAO. When the DAO break happened ETH lost 60% of its value, so not only did 15% of people have their money stolen, but everyone across the board lost 60% of their ETH value. There was simply immense demand for people to get their money back, so people much preferred the fork that kept their money than the fork where the robber stole their money.
Simple as that, it's decentralized, that's the whole point, you fundamentally can't tell people which fork to believe, people use whatever fork they want. And the people mostly wanted the fork with their money in the DAO preserved. People who wanted the unaltered chain stayed there, no big deal.
Wait a second, am I missing out right now? I have an opportunity at Jane Street, but I haven't been exposed to any of the other small name firms. Where do I look to find the higher paying jobs?
We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw), but we found that even better was to calculate Elo scores, do cross-query bias adjustment, and then MSE loss to predict the Elo score itself.